{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "使用transformer解决法语=>英语的翻译任务\n",
    "\n",
    "transformer的核心：self-attentions\n",
    "\n",
    "transformer **优点**：\n",
    "- 无需对跨数据的时间/空间域的关系作出假设\n",
    "- 并行计算\n",
    "- distant item 互相影响彼此的输出\n",
    "- 可以学习到长距离依赖关系\n",
    "\n",
    "transformer **缺点**：\n",
    "- 对于每一个step的xt的输出，是由整个历史信息计算得出，而不再是当前输入和hidden，这可能效率较低\n",
    "- 如果输入具有时间/空间域的关系，则需要加入位置编码，否则整个model也只能看作是一个词袋模型\n",
    "\n",
    "目录\n",
    "* [1.加载数据 建立input pipeline](#)\n",
    "* [2.位置编码 positional encoding](#)\n",
    "* [3.掩码 masking](#3)\n",
    "* [4.scaled dot product attention](#)\n",
    "* [5.multi-head attention](#)\n",
    "* [6.point wise feed forward network](#)\n",
    "* [7.encoder layer](#)\n",
    "* [8.decoder layer](#)\n",
    "* [9.encoder](#)\n",
    "* [10.decoder](#)\n",
    "* [11.搭建transformer](#)\n",
    "* [12.设置超参](#)\n",
    "* [13.优化器](#)\n",
    "* [14.损失和评价准则](#)\n",
    "* [15.生成mask](#)\n",
    "* [16.训练和保存](#)\n",
    "* [17.评估](#)\n",
    "* [18.attention的可视化](#)\n",
    "* [19.总结](#)\n",
    "\n",
    "\n",
    "## 1.加载数据 建立input pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/xijian\n",
      "device= cuda:0\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torchtext\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import random\n",
    "import re\n",
    "from tqdm import tqdm  # 进度条\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "import unicodedata\n",
    "import datetime\n",
    "import time\n",
    "import copy\n",
    "\n",
    "import os\n",
    "print(os.getcwd()) # /home/xijian\n",
    "\n",
    "ngpu = 4\n",
    "\n",
    "use_cuda = torch.cuda.is_available()  ##检测是否有可用的gpu\n",
    "device = torch.device(\"cuda:0\" if (use_cuda and ngpu > 0) else \"cpu\")\n",
    "print('device=', device)\n",
    "\n",
    "project_dir = '/home/xijian/pycharm_projects/Magic-NLPer/'\n",
    "# 当前目录cwd\n",
    "cwd = project_dir + 'NLP/MachineTranslation/transformer/pytorch/'\n",
    "data_dir = cwd + './data/' # 数据所在目录"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(135842, 2)\n",
      "(135842, 2)\n",
      "['Go.' 'Va !']\n",
      "(2,)\n"
     ]
    },
    {
     "data": {
      "text/plain": "     eng         fra\n0    Go.        Va !\n1   Run!     Cours !\n2   Run!    Courez !\n3   Wow!  Ça alors !\n4  Fire!    Au feu !",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>eng</th>\n      <th>fra</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Go.</td>\n      <td>Va !</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Run!</td>\n      <td>Cours !</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Run!</td>\n      <td>Courez !</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Wow!</td>\n      <td>Ça alors !</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Fire!</td>\n      <td>Au feu !</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_df = pd.read_csv(data_dir + 'eng-fra.txt',  # 数据格式：英语\\t法语，注意我们的任务源语言是法语，目标语言是英语\n",
    "                      encoding='UTF-8', sep='\\t', header=None,\n",
    "                      names=['eng', 'fra'], index_col=False)\n",
    "\n",
    "print(data_df.shape)\n",
    "print(data_df.values.shape)\n",
    "print(data_df.values[0])\n",
    "print(data_df.values[0].shape)\n",
    "data_df.head()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "va !\n",
      "go .\n"
     ]
    }
   ],
   "source": [
    "# 数据预处理\n",
    "\n",
    "# 将unicode字符串转化为ASCII码：\n",
    "def unicodeToAscii(s):\n",
    "    return ''.join(c for c in unicodedata.normalize('NFD', s) if unicodedata.category(c) != 'Mn')\n",
    "\n",
    "\n",
    "# 规范化字符串\n",
    "def normalizeString(s):\n",
    "    # print(s) # list  ['Go.']\n",
    "    # s = s[0]\n",
    "    s = s.lower().strip()\n",
    "    s = unicodeToAscii(s)\n",
    "    s = re.sub(r\"([.!?])\", r\" \\1\", s)  # \\1表示group(1)即第一个匹配到的 即匹配到'.'或者'!'或者'?'后，一律替换成'空格.'或者'空格!'或者'空格？'\n",
    "    s = re.sub(r\"[^a-zA-Z.!?]+\", r\" \", s)  # 非字母以及非.!?的其他任何字符 一律被替换成空格\n",
    "    s = re.sub(r'[\\s]+', \" \", s)  # 将出现的多个空格，都使用一个空格代替。例如：w='abc  1   23  1' 处理后：w='abc 1 23 1'\n",
    "    return s\n",
    "\n",
    "\n",
    "print(normalizeString('Va !'))\n",
    "print(normalizeString('Go.'))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pairs num= 135842\n",
      "['go .', 'va !']\n",
      "['run !', 'cours !']\n"
     ]
    }
   ],
   "source": [
    "MAX_LENGTH = 10\n",
    "\n",
    "eng_prefixes = (  # 之前normalizeString()已经对撇号等进行了过滤，以及清洗，小写化等\n",
    "    \"i am \", \"i m \",\n",
    "    \"he is\", \"he s \",\n",
    "    \"she is\", \"she s \",\n",
    "    \"you are\", \"you re \",\n",
    "    \"we are\", \"we re \",\n",
    "    \"they are\", \"they re \"\n",
    ")\n",
    "\n",
    "# print(eng_prefixes)\n",
    "pairs = [[normalizeString(s) for s in line] for line in data_df.values]\n",
    "\n",
    "print('pairs num=', len(pairs))\n",
    "print(pairs[0])\n",
    "print(pairs[1])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after trimming, pairs num= 10599\n",
      "['j ai ans .', 'i m .']\n",
      "['je vais bien .', 'i m ok .']\n",
      "['je suis si heureuse pour toi tom .', 'i m so happy for you tom .']\n"
     ]
    }
   ],
   "source": [
    "# 文件是英译法，我们实现的是法译英，所以进行了reverse，所以pair[1]是英语\n",
    "# 为了快速训练，仅保留“我是”“你是”“他是”等简单句子，并且删除原始文本长度大于10个标记的样本\n",
    "def filterPair(p):\n",
    "    return len(p[0].split(' ')) < MAX_LENGTH and len(p[1].split(' ')) < MAX_LENGTH and \\\n",
    "           p[0].startswith(eng_prefixes)  # startswith first arg must be str or a tuple of str\n",
    "\n",
    "\n",
    "def filterPairs(pairs):\n",
    "    # 过滤，并交换句子顺序，得到法英句子对（之前是英法句子对）\n",
    "    return [[pair[1], pair[0]] for pair in pairs if filterPair(pair)]\n",
    "\n",
    "\n",
    "pairs = filterPairs(pairs)\n",
    "\n",
    "print('after trimming, pairs num=', len(pairs))\n",
    "print(pairs[0])\n",
    "print(pairs[1])\n",
    "print(random.choice(pairs))\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8479\n",
      "2120\n"
     ]
    }
   ],
   "source": [
    "# 划分数据集：训练集和验证集\n",
    "train_pairs, val_pairs = train_test_split(pairs, test_size=0.2, random_state=1234)\n",
    "\n",
    "print(len(train_pairs))\n",
    "print(len(val_pairs))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/torchtext/data/field.py:150: UserWarning: Field class will be retired soon and moved to torchtext.legacy. Please see the most recent release notes for further information.\n",
      "  warnings.warn('{} class will be retired soon and moved to torchtext.legacy. Please see the most recent release notes for further information.'.format(self.__class__.__name__), UserWarning)\n",
      "  0%|          | 0/8479 [00:00<?, ?it/s]/usr/local/lib/python3.6/dist-packages/torchtext/data/example.py:78: UserWarning: Example class will be retired soon and moved to torchtext.legacy. Please see the most recent release notes for further information.\n",
      "  warnings.warn('Example class will be retired soon and moved to torchtext.legacy. Please see the most recent release notes for further information.', UserWarning)\n",
      "100%|██████████| 8479/8479 [00:00<00:00, 99813.09it/s]\n",
      "100%|██████████| 2120/2120 [00:00<00:00, 96628.25it/s]\n"
     ]
    }
   ],
   "source": [
    "tokenizer = lambda x: x.split() # 分词器\n",
    "\n",
    "SRC_TEXT = torchtext.data.Field(sequential=True,\n",
    "                                tokenize=tokenizer,\n",
    "                                # lower=True,\n",
    "                                fix_length=MAX_LENGTH + 2,\n",
    "                                preprocessing=lambda x: ['<start>'] + x + ['<end>'],\n",
    "                                # after tokenizing but before numericalizing\n",
    "                                # postprocessing # after numericalizing but before the numbers are turned into a Tensor\n",
    "                                )\n",
    "TARG_TEXT = torchtext.data.Field(sequential=True,\n",
    "                                 tokenize=tokenizer,\n",
    "                                 # lower=True,\n",
    "                                 fix_length=MAX_LENGTH + 2,\n",
    "                                 preprocessing=lambda x: ['<start>'] + x + ['<end>'],\n",
    "                                 )\n",
    "\n",
    "\n",
    "def get_dataset(pairs, src, targ):\n",
    "    fields = [('src', src), ('targ', targ)]  # filed信息 fields dict[str, Field])\n",
    "    examples = []  # list(Example)\n",
    "    for fra, eng in tqdm(pairs): # 进度条\n",
    "        # 创建Example时会调用field.preprocess方法\n",
    "        examples.append(torchtext.data.Example.fromlist([fra, eng], fields))\n",
    "    return examples, fields\n",
    "\n",
    "\n",
    "# examples, fields = get_dataset(pairs, SRC_TEXT, TARG_TEXT)\n",
    "\n",
    "ds_train = torchtext.data.Dataset(*get_dataset(train_pairs, SRC_TEXT, TARG_TEXT))\n",
    "ds_val = torchtext.data.Dataset(*get_dataset(val_pairs, SRC_TEXT, TARG_TEXT))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9 ['<start>', 'tu', 'n', 'es', 'qu', 'un', 'lache', '.', '<end>']\n",
      "8 ['<start>', 'you', 're', 'just', 'a', 'coward', '.', '<end>']\n",
      "8 ['<start>', 'je', 'suis', 'prete', 'pour', 'demain', '.', '<end>']\n",
      "8 ['<start>', 'i', 'm', 'ready', 'for', 'tomorrow', '.', '<end>']\n"
     ]
    }
   ],
   "source": [
    "# 查看1个样本的信息\n",
    "print(len(ds_train[0].src), ds_train[0].src)\n",
    "print(len(ds_train[0].targ), ds_train[0].targ)\n",
    "\n",
    "print(len(ds_val[100].src), ds_val[100].src)\n",
    "print(len(ds_val[100].targ), ds_val[100].targ)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3901\n",
      "<unk>\n",
      "<pad>\n",
      "<end>\n",
      "<start>\n",
      "3\n",
      "2\n",
      "<start> je suis fais si je vous l examen . <end> <pad>\n",
      "\n",
      "2591\n",
      "<unk>\n",
      "<pad>\n",
      "<end>\n",
      "<start>\n",
      "3\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "# 构建词典\n",
    "SRC_TEXT.build_vocab(ds_train)  # 建立词表 并建立token和ID的映射关系\n",
    "print(len(SRC_TEXT.vocab))\n",
    "print(SRC_TEXT.vocab.itos[0])\n",
    "print(SRC_TEXT.vocab.itos[1])\n",
    "print(SRC_TEXT.vocab.itos[2])\n",
    "print(SRC_TEXT.vocab.itos[3])\n",
    "print(SRC_TEXT.vocab.stoi['<start>'])\n",
    "print(SRC_TEXT.vocab.stoi['<end>'])\n",
    "\n",
    "# 模拟decode\n",
    "res = []\n",
    "for id in [  3,   5,   6,  71,  48,   5,   8,  32, 743,   4,   2,   1]:\n",
    "    res.append(SRC_TEXT.vocab.itos[id])\n",
    "print(' '.join(res)+'\\n')\n",
    "\n",
    "TARG_TEXT.build_vocab(ds_train)\n",
    "print(len(TARG_TEXT.vocab))\n",
    "print(TARG_TEXT.vocab.itos[0])\n",
    "print(TARG_TEXT.vocab.itos[1])\n",
    "print(TARG_TEXT.vocab.itos[2])\n",
    "print(TARG_TEXT.vocab.itos[3])\n",
    "print(TARG_TEXT.vocab.stoi['<start>'])\n",
    "print(TARG_TEXT.vocab.stoi['<end>'])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([   3,    5,    6,   69,   75,   15,  108, 1951,    9,    4,    2,    1])\n",
      "torch.Size([12, 256]) torch.Size([12, 256])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/torchtext/data/iterator.py:48: UserWarning: Iterator class will be retired soon and moved to torchtext.legacy. Please see the most recent release notes for further information.\n",
      "  warnings.warn('{} class will be retired soon and moved to torchtext.legacy. Please see the most recent release notes for further information.'.format(self.__class__.__name__), UserWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/torchtext/data/batch.py:23: UserWarning: Batch class will be retired soon and moved to torchtext.legacy. Please see the most recent release notes for further information.\n",
      "  warnings.warn('{} class will be retired soon and moved to torchtext.legacy. Please see the most recent release notes for further information.'.format(self.__class__.__name__), UserWarning)\n"
     ]
    }
   ],
   "source": [
    "BATCH_SIZE = 64 * ngpu\n",
    "\n",
    "# 构建数据管道迭代器\n",
    "train_iter, val_iter = torchtext.data.Iterator.splits(\n",
    "    (ds_train, ds_val),\n",
    "    sort_within_batch=True,\n",
    "    sort_key=lambda x: len(x.src),\n",
    "    batch_sizes=(BATCH_SIZE, BATCH_SIZE)\n",
    ")\n",
    "\n",
    "# 查看数据管道信息，此时会触发postprocessing，如果有的话\n",
    "for batch in train_iter:\n",
    "    # 注意，这里text第0维不是batch，而是seq_len\n",
    "    print(batch.src[:,0])\n",
    "    print(batch.src.shape, batch.targ.shape)  # [12,64], [12,64]\n",
    "    break"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [],
   "source": [
    "# 将数据管道组织成与torch.utils.data.DataLoader相似的inputs, targets的输出形式\n",
    "class DataLoader:\n",
    "    def __init__(self, data_iter):\n",
    "        self.data_iter = data_iter\n",
    "        self.length = len(data_iter)  # 一共有多少个batch？\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.length\n",
    "\n",
    "    def __iter__(self):\n",
    "        # 注意，在此处调整text的shape为batch first\n",
    "        for batch in self.data_iter:\n",
    "            yield (torch.transpose(batch.src, 0, 1), torch.transpose(batch.targ, 0, 1))\n",
    "\n",
    "\n",
    "train_dataloader = DataLoader(train_iter)\n",
    "val_dataloader = DataLoader(val_iter)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "len(train_dataloader): 34\n",
      "torch.Size([256, 12]) torch.Size([256, 12])\n",
      "tensor([   3,   11,    7,   17,   46,   10, 1261,  117,  143,    4,    2,    1]) torch.int64\n",
      "tensor([   3,    9,   15, 1097,   53,  405,    4,    2,    1,    1,    1,    1]) torch.int64\n"
     ]
    }
   ],
   "source": [
    "# 查看数据管道\n",
    "print('len(train_dataloader):', len(train_dataloader))  # 34 个step/batch\n",
    "for batch_src, batch_targ in train_dataloader:\n",
    "    print(batch_src.shape, batch_targ.shape)  # [256,12], [256,12]\n",
    "    print(batch_src[0], batch_src.dtype)\n",
    "    print(batch_targ[0], batch_targ.dtype)\n",
    "    break\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 2.位置编码 positional encoding\n",
    "绝对位置编码\n",
    "\n",
    "由于model中不含有任何recurrence or convolution，所以句子中token的相对位置关系无法体现，\n",
    "所以就需要在embedding vector中加入position encoding vector（维度相同）。这样每个词的词向量\n",
    "在 $d_{model}$ 维的空间中，就可以基于meaning和position来计算相似度或相关性\n",
    "\n",
    "$$\\begin{array}{ll} & PE_{(pos,2i)}=sin(pos/10000^{2i/d_{model}}) \\\\ & PE_{(pos,2i+1)}=cos(pos/10000^{2i/d_{model}})\\end{array}$$\n",
    "\n",
    "**特点：**\n",
    "- （1）后面位置是前面位置的线性组合，保证了即使位置不是相邻的，也可能有关系（[参考这里](#https://blog.csdn.net/zhulinniao/article/details/104462228/)）\n",
    "- （2）每个位置的编码又是独特的\n",
    "- （3）每两个位置的encoding互相做点积，位置越远，点积的值越小，自己和自己点积，值最大\n",
    "\n",
    "![jupyter-img1](./imgs/im1.jpg)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 50, 512])\n"
     ]
    }
   ],
   "source": [
    "# 计算角度：pos * 1/(10000^(2i/d))\n",
    "def get_angles(pos, i, d_model):\n",
    "    # 2*(i//2)保证了2i，这部分计算的是1/10000^(2i/d)\n",
    "    angle_rates = 1 / np.power(10000, 2 * (i // 2) / np.float32(d_model))  # => [1, 512]\n",
    "    return pos * angle_rates  # [50,1]*[1,512]=>[50, 512]\n",
    "\n",
    "\n",
    "# np.arange()函数返回一个有终点和起点的固定步长的排列，如[1,2,3,4,5]，起点是1，终点是5，步长为1\n",
    "# 注意：起点终点是左开右闭区间，即start=1,end=6，才会产生[1,2,3,4,5]\n",
    "# 只有一个参数时，参数值为终点，起点取默认值0，步长取默认值1。\n",
    "def positional_encoding(position, d_model):  # d_model是位置编码的长度，相当于position encoding的embedding_dim？\n",
    "    angle_rads = get_angles(np.arange(position)[:, np.newaxis],  # [50, 1]\n",
    "                            np.arange(d_model)[np.newaxis, :],  # [1, d_model=512]\n",
    "                            d_model)\n",
    "    angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])  # 2i\n",
    "    angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])  # 2i+2\n",
    "\n",
    "    pos_encoding = angle_rads[np.newaxis, ...]  # [50,512]=>[1,50,512]\n",
    "    return torch.tensor(pos_encoding, dtype=torch.float32)\n",
    "\n",
    "pos_encoding = positional_encoding(50, 512)\n",
    "print(pos_encoding.shape) # [1,50,512]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 432x288 with 2 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEKCAYAAAD+XoUoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABfiklEQVR4nO2dd3gc1dm372dmd6VV78W9N5oxxmAceu+EUBMCJCSkQAIJCYHkg/QE8r6BNAihBfImgVBCAgRiOqZjA+7dcpVsWb1um5nz/TGzq5UsWStbsiXr3Nd12Jkzs2fOmNXZ2d/TRCmFRqPRaIYHxv6egEaj0Wj2HXrR12g0mmGEXvQ1Go1mGKEXfY1GoxlG6EVfo9FohhF60ddoNJphxIAu+iKySUSWichiEVnk9RWIyMsiss57zR/IOWg0Gs3+QkQeFpGdIrK8h+MiIr8TkfUislREZiUdu8pbJ9eJyFX9Nad98aR/olJqplJqtrd/C/CqUmoy8Kq3r9FoNAcijwBn7Ob4mcBkr10L/BHch2Pgh8BRwBzgh/31gLw/5J3zgUe97UeBC/bDHDQajWbAUUotAOp3c8r5wF+Uy/tAnoiUA6cDLyul6pVSDcDL7P7LI2V8/THIblDASyKigD8ppe4HSpVS273jO4DS7t4oItfifvORmRE8ol1lMHP6WBav2szMaWPY+skKxh40nsVbmsjMz2VUy3YaGiOUHX4QS9dtxx/M4KAiE2VbrGnx0d5QR/GIUkaqJqo21pBuCEXTxrGxTWjYWYfpD1BUnEdNdR3KccguLGBiYZDI1grq60LYCvIy/GSNKaXdn8Om6hZKCzIoTAOrZgdtO1tosRwAMkyDzLw00ooLsYO5VH6ygoAImWkm6XlB/Pn5OOnZtERtGtqitIcsrEgYOxYFx2bWxGKctmaiLe3E2qJEow4RR2ErhQMIYAr4RCgcmYcVimCFLeyITdRxsBwS58bjrQ0g56BpRG1F1HKIWjZRy8GxldscB+XYXnO3DysLID4/YvrBMFGG6b4iOApsBUq581pTsR0RAREE79UwOvYNAxEDEcGfZqIUoBTKG8PdB+X+h3ikuFIOWdnpiAgCGCJ4l0EQDME95vVVVdYl7lq5A3T5RHbsTxxfjsQ/b95/xNvrvO+yav22VD7zABw8eXS3/SK79i1bsyXlcQEOnTam+7G76VuyOvWxZ/Ywbncs7sO47thj+zD25tTHnd553MWrNqNCdbVKqeKUB+mCkTNKYYVTOleF6lYAySff761zqTIS2Jq0v83r66l/rxnoRf9TSqlKESkBXhaR1ckHlVLK+0LYBe8f7n6AIw49SC0zj+Kdd+4ld+7XWfD2PXwnczr3PPUAhdfP55iLzuKXr/2UZ55bx/feeYcR5/2SsoOO4INrMrGb6jj+zUI+evJvXHb7t/hl9Hl+/PkHmJIV4OqnHuDzH6bzzD2PkFU2jquvPZs/3v13YuE2jr3yEp763MFsvOHz/O2vy2iKOVw4vYxjfv8dlow8iSvvfoubLj+MKyeY1N73cz74wwJer2kHYFZuOkedO5mJ115Fy8Fn8oOcGYxI8zF3XC5TLziUERddRNu0k3hzcxOPL9rK0qXV7NywltYdm7DCrXz45LW0f/ASlW8upmphJZu3NLOpPUZ91CbqKEyBXL9JUcDkqpvPp3bpBurW1NJQ0Uhla5SaiE1DzCZkO9jev27AEM586iW2NIXZVNvG5ro2quraaWuO0N4UIdweJdLSSLS9CSvUihVu453vjMEsLMPML4HMPJy0bJxgHjEzjfaYQ1vMIWQpmiMWJ132I0x/AMMXwPD5MXwBzLQgpi+Q2DZ8AXwBP6MmF2JFHayYjRWzsS0HK+bgWA627WBbDo7tYFsWjhVl7glTCfgMAj7TfTUN0nyG19e53fbDR1CO7X6GvC8vd9t9dbxXgHv//H0MAVMEQwTTcL9Uuu6LgIFwxHk3dxprdzz30t1AxyIf/0ktXoeRtEKPPfEbvY6XzKtv/qHbBd7oprPk2OtTHvfNt+/ptN/dNeIUzLsu5XEB3n7n3pTPzTvm6ymf+06XcXPnfp3Y4j+n/q3RHVYY39TzUjo1tvjP4STpekgwoPKOUqrSe90JPIOrTVV7P1/wXncO5Bw0Go2mT4gghplS6wcqgeSfhaO8vp7695oBW/RFJFNEsuPbwGnAcuBZIG6Jvgr490DNQaPRaPqOeL9Ye2/9wLPAlZ4Xz9FAkyd/zwdOE5F8z4B7mte31wykvFMKPOP9nPUBf1dK/VdEFgJPiMg1wGbgkgGcg0aj0fQN70m/f4aSx4ATgCIR2YbrkeMHUErdB7wAnAWsB9qBL3jH6kXkp8BCb6ifKKV2ZxBOmQFb9JVSFcBh3fTXASf3ZayVO6PM/e6VvDHtKOZ+47e8f+RxXHJICZe8637TPntuPjd8fRX/7+dnc+YfPyDcVMtfvnUsr595GrGnn2fpC3cwZu453Hn6BF6d+hghW3HqV+byQfoM3nzxaZRjM+O4I3nqhTW011Ux9phzufmUKTgvP8iSZ9dSE7GZlZfO9EtmY808m7/+dx2HH17OaZMKcT78O5teXsGypghRRzE66GfCpHxGHjcTJh/FypoQWT6D8Zl+Sg4poWj2Qagxh7ClOcpHWxvZuK2Z5toGwg3VWOFWAKIVK2hcu5XGjQ00bm+lJmLTajlEHVegDxhCpmlQEDBpq6ylfWcr7bUhmsIWrZZDm+2eG9fzTXFbQyhGQ3uUurYoda1RIiGLaMgiGrGIhduxoyHsSAjHiqIcGyMjGyM9EwkEcXzpqEAGypdG1FKuQdhRRG2HiOUgZvwnr4EYJoY/gOH9BDZ8AcQwMX0+RAQ7rt3bDspxDcnKUTjKfVVK4TgqoZ2bhmAahvsq4u1305KspMpxdv/5tL2xU9TzO8btXc/vC90ZdveE7vR82YvB+2laQxIBxOyfRV8pdXkvxxXQrYFEKfUw8HC/TCSJgTbkajQazdBCBKOfnvQHI3rR12g0mi70l7wzGNGLvkaj0STTj5r+YEQv+hqNRpOEIBg+//6exoAxJLJsRloaefWkMC9ta+a1s02eWVXD0R8s4Pl7HuRnP72GV477LEfmB6m9+hd88PgTzLnkYma8cw/PrKrhxnveQ9k2t11zJLV33sgLlc2cOTqH8m//mO89uZTatQspOWget513EJUfv05m8WjOOmUSR2c0suL+51nYECbXbzBr7kiKL/wcr2xs5M2FW7ls9mhGtm2k8sXXWLu8huqIRdAUZuQEGHXMODKPOokdRh7vbK6nNM3H6HF5lM2eRPohc2kIFLJ4ewsfb26gfnsLbTu3EG1rAsAMBAlv2kDj+iqatzVTE7FpttxAK3CNuFk+g1y/Z8jdUUdrdRvt9SGaYk7C4GsnRZ6aIgQMoT4cY2dzhLrWCOFQjGgoRjRiuQFSkRBW1DXiOlYMOxbFyMzByMzGCQRx/EGUL42YwjXgOgrLhvaYTdhyEkbbuBFXko24ZtyYK5g+A+XgGmwdhW07ntFWJYKzlGfEVY6Nsu2EoTZgugFY8cCsroZcQ6SToXV3gVnQvfGzJ/piE41fL5XArD1hOBtZ9wn71k9/n6Of9DUajaYLQ3VBTwW96Gs0Gk0yIv3msjkY0Yu+RqPRJCEc2E/6Q0LTHz2mnF8ccz23//5S7pp9Dd/97vEc+YOXKT/8FK6p/hf/qmjgs8/+mAt/9hoZhSN4/mtH8fh1f2VKVhob336WQ88+jysKanjut29REDA57pcX8+hGxYpX3yKQmcupZxzMCRm12NEQE46ayw3HjqPxsT/w/jvbaLUcji4IMv3zJ1JVcDAPv7uJqlWrOXFcLqEFz7DxlQ2sbY1iKxiXEWD0rDLKTzwaa+wRfFTVwuurdjIpy0/5rHJyZ84kVn4Q6xvCLNrcQNXWJlp2bifa2oBjRRHDJJCZS8ParTRtbqa+LpQIzEpOnBYPzMooCNKyvZX2uhD1UZummE3Y09uTA7MChhA0Depbo9S3RWmMB2ZFbGIRCyvUih0N4cQ8PT8enJWZjQpkovwZ4E/H8aURjgdm2Yqw5RC2HNpjdqdEa2KYGElBWYYvgGEIpmlgmkYiMMu2HJRDQstPaPtxTd92df14ojXTEHxJGn4nXV8E0xO7+zswSxLjph6Y1ZOe3905e4sOzOpnxMD0BVJqQxH9pK/RaDTJyIH9pK8XfY1Go0lC0H76Go1GM6w4kBf9IaHp5zRUUpbu494pXwRg6VV3su71Z3jtjrP47efu5crjxvBbaxab332Ob990MVU3fo6FDWEuv/0MckZN4ZEvz+GT677LkqYw5580jrazvsWvH1tCa/Umxh19Iv/vlEls/+P/UjhpFl87dzpjKt9jyYNvs6olwuign4M/PZ3AKVfyrzU1rPhkO83b1hJc9xYb/v0eS7c0UR+1KQiYTCvLZPQJMwgcfiLrmxVvrKulsqKBEYeUUHbUDHwzjqYyYvLx9maWbKqnvrqVUMOOhI++L5hFWm4Rjeurad7WzI5w3Ee/I9Fals/V83PTfWSWZtBW3UZLU4SmmEPYUYTsjsRs8fcEDCHdkISPfiRkEQnFiEUsYuEwdtT10bc9P/24li7BbFQgiPKn4fiDRCwnoedHbUV7zKY95hCxnUSitXjitYSe7/nsG6aB4TMQQ3Ast2CKUsotmNIl0Vo84Vu89ZpozfPRNwxJ6Pm9+ejH6UnP70qqvvW96f7xcQarnr+/GRRT1376Go1GM5zQ8o5Go9EMG0QEwz80PXNSQS/6Go1Gk4xOuKbRaDTDC73o72d2VLfyhR2LyDnvf2n98H6KbvwjJ375Gtq+cSlttsOsF1/k7At+xcQTLuCW8iq+/8hiPjOtkPAXf85l4ysYs+A+fvzKRo7MT+fwu37E1c+tYtO788kdM53rP3MwI1f9h2cfeJ+ZP76GKw4uYsON3+LtigZMEY6ZWsDYKy5hcSibx978hJo1H+FYUXY+9wwVC7awNRQjYAhTsgKMmTeK/GNPoDFvIm+vrGHRmhoaKqsonz2OzJlzCRVMYPmmJt5dV0ttZQttNVuItDS4gVC+AIGMHDIKR9L4URPVLVEaYh1GXFMgy2eQ4xlyM0szySrNpH5dA/VRN4AruboWdDbiBk2D+rYILW1RIuEY0ZBFLGJ1JFrzArOcJAOqCrhJ1pQ/AwuDqO14zTXiRmyHiGW7lbOSEqyZXYy4ps/A9BmuodRnJAKxbEslEq/FA7OSE611MuR2CczqFKDlBWbFK2ftzogbD8yC7g22cZIDs/bUiNvfidb2BUNgivsEYyj8z9pDhoT3jkaj0ewrRAQxUmspjneGiKwRkfUicks3x+8WkcVeWysijUnH7KRjz/bH/Q2JJ32NRqPZl5hm/zwPi4gJ3AOcCmwDForIs0qplfFzlFLfSjr/G8DhSUOElFIz+2UyHvpJX6PRaJIR+vNJfw6wXilVoZSKAo8D5+/m/MuBx/rhLnpkSCz6pYVBDv+fFYw84mROni+YaUFePAXueXwl37nnco7/33exwm3869YT+O/p3yRgCCc9+Ssuve8D7jqphBeuf5Soozjruyfzikzl5WfeAeCwU+fyhWmZLP3lAyyobeenZ8/A/vfdfPjPVewIW8zKS+eQL3yK9sPO4YH3N7PxkzW011URzC9j/XNLWNIUIWQrRqT7mDy9iNGnzIZp8/hkRxsvrdhB9ZZGWqs3UTz3cJzxh7OhIcKHmxvYsKmRpupawg3VWOFWAAKZuQTzy8guyKJxeys7wlYnjT5oGolEa7kF6WSVZJBZlkdT2KIp5tBmO7skWktOtpblE+riidZCFtGIRSzcjh0NYUdCiYAoJ9YRGOX4M1CBDBx/OpF4UJajiNoOES/RWvw1ruEbRufgLNPnwzTdoCw3OMstoOLYnpbvBWjFA7OS9XxwdXJTpMfCKfGgKsML0NodyXo+9ByYFdfzk+lr0NDu/rCSx9qbP8ADLdHaoAjMIp5ls98W/ZHA1qT9bV7frtcVGQuMB15L6k4XkUUi8r6IXLBnd9QZLe9oNBpNJ3p/gEiiSEQWJe3fr5S6fw8vfBnwlFIq+elkrFKqUkQmAK+JyDKl1IY9HB/Qi75Go9F0xpN3UqRWKTV7N8crgdFJ+6O8vu64DLguuUMpVem9VojIG7h6/14t+kNC3tFoNJp9ST/KOwuBySIyXkQCuAv7Ll44IjINyAfeS+rLF5E0b7sImAes7PrevjIkFv1Q6VjWv/k8y+46i3f/8ijP/v7LPHjUNXxmWiEvzv4anzzzGJdffwWZ99zEc9ua+eL1x3B/60QW//ufrL/hy7yys41PH1FO7o2/5pa/fER9xRLGzDmV337mUBof+CmvvLkFgMOja/noNy+wsCFMWbqP2WdMIO8zX+Kfq2t5670tNGxajuELUDBpFivW1LMjbJHrNzikIMjYk6eRMfcsNscyeXVtDRvW19G4dS3hphoChx7HDnL4YFsT762rpW6H66OfSLSW7iZayywqI684g8qQRbPl7FIMvSBgUpTmI7Mkk6wR2WSNLKY+atNmO7skWjPF1fLTDYMsn9va26JEQzEv2VoUK9S6SzH0uJ4PeMnWgh1FUzolWutooZjdkVjNF3Cb33v19HzTNBKFVBJ++nbnxGvJSdaSW7KWH+ii7RtJPvqmpJ5oTTl2r4nW9sZHv2OMnn30+1vPH8oMFj0f3LmYPkmp9YZSygKuB+YDq4AnlFIrROQnInJe0qmXAY8rpVRS33RgkYgsAV4H7kj2+tlTtLyj0Wg0XejPTKVKqReAF7r03d5l/0fdvO9d4JB+m4iHXvQ1Go0mCfG8wQ5U9KKv0Wg0XeiDIXfIoRd9jUaj6cKBvOgPCUPups07+P4vvsUb045i7hVXUviLL7OpPcbx78/n+v/3f4yZew73zbb4052vcf7YXIK33cdP736RQGYujz2xksNy0znmvtu58bnVrHntRXJGTeHrlx3KlC2v8f6vX2VTe4x5hUEq7v5f3li6E4DjJhcw+cufZaWM4OFXN7BjxUfY0RA5o6Yw8dAy1rZGMAWmZAUYf+JYSk45meaSGby5qZ63lldTs3Eb7bVVKMcmVDKVpdVtvL2uhp3bmmnevolwUy2OFcXwBUjLziejcCR5JZmMK8+m1kugZis3wCpoCjk+g+I0k8zSDLJHZJE1spjMkcU0xboz4rrvSfcMwFk+ISvNR7g9RsRLtBY34tqREHY0jN2lWhWA8mcQEx8RyyHsVc0KxxzaYx2BWWHbIRS1vUCsXROtxY23ps/AMN0ALdtSrgG3h0RrQKd5dJdoLVFNS0gEZsV/kndnVE0OzNpddavkRGtd+3qiL0bcgTRY7quKWX3wYR+aiHuPqbShiH7S12g0miQE9+HkQEUv+hqNRpOMHNiplfWir9FoNF0YysXle2NI/IbxZ2Tz9VX389K2Zl47E+5+4GNuue9zzPvNx0SaannxR6fywqe+gCnCaS/8lgt+/x61axdy3GXn0mo5XPj9U3k5eDjP/uNNlGNzxJnH8rWDsljy49/zys42Rgf9HPWFI3n7sWVUeYnWDrv2eEKzP81vF1RQ8fFq2mq2EswvY/TBM/j83LGEbMXooJ/ph5Qw9syj4JCTWLS9jeeXbmf7xnpaqzdhhVsxfAHWN0R4p6KOtRUNNFTu6DbRWk5RLsWlWRw6Om+XRGs5PjORaC27PIvMsjyyRhbjKx7pBWZ1TrQWL54ST7SW6zdJy0kjGrLcwKykRGt2NLxLorU48URr4aREa/GArHiitVDUbQk9v0uiNcNnJBKtxQup9JRoLXkOiaRvXYKzEkFappHQ8f2G4Wr7Xf5Q44FZPen5vSVaM6R/NfjBmmhtfzPYpu4mXEutDUUGfNoiYorIJyLyvLc/XkQ+8AoK/MMLTdZoNJrBQdw5IIU2FNkX31U34IYfx7kTuFspNQloAK7ZB3PQaDSaFBEM00ipDUUGdNYiMgo4G3jQ2xfgJOAp75RHgQsGcg4ajUbTF0Q/6e8VvwFuBhxvvxBo9JIQwe4LClzrFQ9YVJoW5oc3Ps0P772cu+Zcy+eOHslfpn2BJf96nBtuvQb5yTU8v72Fr9x+OndsH8Hifz/FhOPO54krD+eyE8fh+9qdfPfBhdRXLGHCvDO45+JDqfntbbzw+mZMgVPmjWL017/NwoYQo4N+jv70VHIu/jqPLd/J2+9spmHTcsxAkOJpsznt6DGcOamAgoDJzLIsxp9xCOnHnMv6cDovrKxmw9o6GresJtxUgxgmwfxS3tvayPvraqmtavaKodcDbqK19PxSskvKKSjN5OCRuUwtztol0Vpxmklxhp/MkkyyR+WSPaaUtLIyfGVjdvHRj2v5maabZC3XbxLI8JOWEyASihENhbBCrcTCrV6iteguidbiuP75bpK1iKVoieyaaK01bNEetXebaM30ea+exp9qorV4kfZUEq0ZRueEaz0lWkumt0Rr8e6ufvvJ7G2itf7Q4velnt/fvumDTc+P0581cgcbA7boi8g5wE6l1Ed78n6l1P1KqdlKqdlFhYX9PDuNRqPpHhG6Dwbspg1FBtJlcx5wnoicBaQDOcBvgTwR8XlP+7srKKDRaDT7haG6oKfCgD3pK6VuVUqNUkqNw80V/ZpS6nO4eaEv8k67Cvj3QM1Bo9Fo+oqQ2lP+UP1i2B/BWd8DHheRnwGfAA/thzloNBpNt4hAQKdh2DuUUm8Ab3jbFcCcvry/dvkaLp51OPdMvJoATzH5vy9x1rk/5JBzLuG24Mfcct9CPnf0SOqv/iV3XfN7ssrG8fC3PkXld65k9oO/4dy/LWb9m89TOGkWP7z6CEYt/CtP/n4BVWGLc0flMPPWL/CuPYqAIZwwq4xJ132Vd1qzeXj+x2xf9h52NETRlCM5bPZIrpg1iqLqxRyck8bE0yZSfNpZ1ORN4uXl1by7bAe1GzfSXucmWkvLLiCrdDyvrKxmx+ZGmrdXEG6qdY2TgSDpuUVkFo8hvzSLqaPzOGRkLpMKMnjBS7SW5TPI95sUp5lkj8giZ5RbLStzRAm+0jGQW7KLETdgdCRay/UbBAMm6fnpBPPTiYRiHdWyYlE30VrMNeZ2Z8h1K2U5RKxdq2W1JQVmhWJ2JyOu6XMTrMUTrYlIIkjLNI1dEq05VhRldzbiQkfStU5BWT4jYbz1JwVoJSfASjbi7kmiteQHuD1JtJZ4bzeJ1vrbiJvq9ftnvGFixBU3yd+Bik7DoNFoNEkIB7amrxd9jUajSUaGrl6fCgeucKXRaDR7gPukb6TUUhpP5AwRWeOlnrmlm+NXi0iNiCz22peSjl0lIuu8dlV/3N+QeNK3FYz970ucefYttH54PxNvep7M4tG8+7253DdiDtOz0zjqxWc45LbXaK+r4uaffIOZH/2ZXz30MTmXZvHuU08SyMzlks8ez2dydvLmLX/mnboQh+Wmc/T3Tqdm5oXc/pePuakki8O/dT5bR8/jzqeWsXHRx4Sbasgun8iEWdP40jHjmGLUUfvsE0w9eiSjzz0Za8ZJLFjXwLMfVbK9Yiet1ZuwoyF86VlklY6jaEwpFRvqaaqqpL2uCjsaQgyTQGYumcVjyCvOZOSIbA4dncu0okzKMt3/Jcl6fm5JJjmjsskZU0L2mFLM0jEYJWOws0t3SbQW9IKysnwGOX6ToKfnp+enY4VasaMh77X7winJRCzlJlyznCQ93yFiuYVT4oFZ8SIqhi/QUTQlnmzNlIS+bxiC6RNsy8GxHWzLShRO6S4wK06nhGsi+I0OHT+eaM2UXX+S707Pd20FnROt9VQ4pavO31f2R+GUwa7nD3b660lfREzgHuBU3GDUhSLyrFJqZZdT/6GUur7LewuAHwKzAQV85L23YW/mpJ/0NRqNJglDOiLAe2spMAdYr5SqUEpFgceB81OcyunAy0qpem+hfxk4Y49uKgm96Gs0Gk0XXA+x3htQFE8X47Vruww1EtiatN9T6pnPiMhSEXlKREb38b19YkjIOxqNRrOvkG6kwt1Qq5SavZeXfA54TCkVEZGv4CaiPGkvx+yRIfGkX3bQBOZ+5WFGHnEyJ88Xqpct4Lm7r+SdY06lKhzj6ud/wumPrGbj289y1GWX8MPJrTxx7UM0xWx+fe+rhBqqOfzcM7nz9Aksv/lWXlhVS1m6j1OvOJSsq/8fP39tA6ve+oQjvnEcnHU9v3lrE0vfWkXztrWk5xYz6tDD+fwJEzhxTBbR1/7Gun99zOQL52LMOZeF29v51+JKtqyppWnLSiIt9Ri+ABlFI8gbNYYJEwuoq6yltXoTsbYmwC2cklE4gtzSIkpG5jBrbD4zirMYlRMgK1LvFUJ39fzC/HSyRmSRPSqf7DGlBEaOxT9iHE5WEZFANpCs5wuZpuufn+s3SMsNkO7p+en5mVjhVmKhjkRr3RVOiSOGSdh2C6K3Ri1aPX/89phNa8Tq0PNjNqGoheEPYPp8bsrZuE++Tzr56xumm7I2uXDK7hKtxfX+RMK1JL/8ronW4n763RVO6YlUCqf0NdFa8jhd37+vEq0NBceTwW4i6MeI3EpgdNL+LqlnlFJ1SqmIt/sgcESq790ThsSir9FoNPuKeHBWKi0FFgKTveJRAdyUNM92vp6UJ+2eR0f9kfnAaSKSLyL5wGle316h5R2NRqNJQpB+S8OglLJE5HrcxdoEHlZKrRCRnwCLlFLPAt8UkfMAC6gHrvbeWy8iP8X94gD4iVKqfm/npBd9jUajSaKPmn6vKKVeAF7o0nd70vatwK09vPdh4OF+mwx60ddoNJpOHOhpGIaEpr+yJkakpZ5ld53Fu395lJt/8g3yfvllnli2k2//7GzuaD+M9/72dyYcdz7//eoc3rjg67xfH+LyU8ZTs/p9Jp94Pn++6ghq77yRfz+3Dlspzjp+DOO/dxsPLKvnhRdWUF+xhKJrvsvDi7fzwivrqV27EDMQpGTGUZx7/Hg+Pa0IefcJ1v5jAUuX15B50mdYb+Xw1JIqli3fSf3GlYQaqhPVsvJGT2HE+HxOnF5CS9X6TtWygoUjyCkbRWF5FrPG5nNIeQ7j89IpMCL46jeT6wVlFWf4yRmVTe6YPHLGlZM+ejT+EeOwc8qws4ppCLvGxO6qZWXkpJGe5wVm5QVJy8smFnaDs+KJ1nZnxBXD7LZaVlt0VyNuonKWmZxorYcgLZ/RqVpWsjG5OyNucsK15GpZ8QAtf7zfM+h2R3eBWbvc826qZRlCp7RrvRlxk8eM05MRd0/XFl0tawDRRVQ0Go1m+BDPp3+gohd9jUaj6YJe9DUajWaYYBzgRVSGxJ2Fmxt56cEbeWPaUcy94kpurn+K3/xpEV+9cCrLPn07v/7loxRMOIx//+BE1n3xMzyxbCcXTMhn1sN/pOywE/nNV46i9OXf8txv36IqbHHW5AIO/+mNvNhawh+fXE71sgX4M3N5sTadB59bRdWSBSjHpmjKkXzqU+O46ohRFGx6h01PPMfyt7eytjVCZfZEnl1VzTuLq6het4a2mq2Jwik5o6ZSNjafkw4qZe6ofEIN1YnCKcH8UrJLx1I0ModDxhVw2KhcphZlUJ5h4KvbRLRiBUUBk7J0n6vnj8ohZ3w5mWNG4i8fh8ofgZNdQkPEoTFsJwqndOj5BplBX6dEa8HCXNILc7AjIRwrttvCKXE9XwwzoeO3RDsKp7SGLTfZWsSiNRxLJFyL6/U+v9mRYC2pcEpC6zekx8IpXfX8OAGfgd8weiycYhqdi6j0lmgtca+7KZySrOf39P5U0YVTOhj0ej5oTV+j0WiGE0Iir84BiV70NRqNpgsHcippvehrNBpNEgI9uv8eCAyJRX/U6DKM6y7hpW3NvHYmfH/mw5w3qYCCB57mjC8/hBgG99x2AZn33MR9T67i6IIgpzz1S360xOEHXzuO43a+zn++9RhLmsKcUpLJp+68ihUjjuPHD37Ipg9fQwyTMUeeyJ3PrmTTh+8Sa2uiYMJhzJg7mW8cO4EJbeuofPwx1jy3luXNEUK24sV1dTz3wVa2r91M645NOFYUf2YuOSOnUDaumGNmlPCpcQVMLUzDsaIYvgDpuUVklY2noDybyWPymDU2j4NKshiZ5cdftx5r43La1q9z9fyR2eSNyyVnfBk548rxjRiPFI3Cyi6lyTJoCNtsb4kkkqzF9fzcdF8iyVpGUQbpnp6fXpjr+ujvpnBKsp4vhklr1KY1anUkWgu7PvotESvhnx+K2lgx29XykxOrxTX8JJ99n5eDPK7nO70UcUkURk9KrmaI4DelU+GU5O1U9XzYVc/vLvkauIuAIdInPb+7B8Wuen5/++gPdj1/yOB91g5UhsSir9FoNPsKAfwplkIciuhFX6PRaJLQ8o5Go9EMJzyX4AMVvehrNBpNEnEbzoHKkBCu8pq288Dz6/jhvZdz15xrmZ6dxgkfv86J359Pc9UGfnDb1Zy65AH+dOdrjEj3c+mfv8Zf7Bn86b7n+XJRNW9+6U7mV7cxKy+dU356PjvnfZEbHl/M2jffwAq1UnbYiVxxzjRWL3iP9roqsssnMvnoQ/n2yZM5zFdD7ZN/ZuUTi1nYEKIp5lCcZvL4e5vZurqSxq2rsMKt+NKzyCmfSOmEkcyaUcKJk4s4uCSD4M41iGG6RtzS8RSNLGDC2DyOmlDAYaU5jM72k964BXvLKkLrV9O4bisFpZnkjc0hd1wpuRNH4h81CbNsPHZuOW2STkPEpqolQmVLmGCSETc/4AZlZRRlkFEYJL0wO2HE9eUVYHvVsuIG1O6IG3FNf4CWiFsxq8VLspZItBa1E6/RqI1tObsmVosHZPncalm+pGLSXYOyekq0Fm8dydWMRJWs5ECtjspZHfeRapK15O24EbeTcXfvPro9/oH1v9G1v8fr/0VvKK2jbmK/3ttQRD/pazQaTRLiPVAcqOhFX6PRaJI40OUdvehrNBpNF4aqdJMKQ+I3zPYdLXzvpmO5Z+LVAFyx5Cnm/Owdti38L1d864t8036XP1z7f5gifPmui3hjyqXcfvfLNG1ZxXtX3cS/1tQxJSvAuTefjH35/+P6p5ex7OUFhBp2UDJjHuedOZXrjhpFy/YNZBaPZtLRc7jxjKmcWGzR8q+HWPHXD/iwqoWaiE2u32BWXjobl2+ncdNyYm1NmIEgWWXjKJk4gUOml3DatBIOL88it2kj0eXvkJ5bTFbpeApHlzB6bB7HTC7i8PIcxuUFyGyrRm1dRXjtchrWbqVhXQ35E/LIHV9C7qSRBEZNwDdiAnZuGe2+LOpCNjtaomxvibCtIUSOz6AgYFIQMN3kakVBMoqCBIuySS/MJaMkH39+PkZO4W71/OSgLNMfSARndQrKClu0RixawrGEnm/FbKyYg+Ez8Pk7J13z+Y1EYZW4np/mMzoSrvWi58dJ1vN9ppGk4Xfo+X6zI19KKnp+YmzpXc83RPZIj+7vwik9XmcILFBD6cFZ6Ejg11tLaTyRM0RkjYisF5Fbujn+bRFZKSJLReRVERmbdMwWkcVee7bre/cE/aSv0Wg0yfRjjVwRMYF7gFOBbcBCEXlWKbUy6bRPgNlKqXYR+RrwK+BS71hIKTWzXybjMSSe9DUajWZf4Wr6qbUUmAOsV0pVKKWiwOPA+cknKKVeV0q1e7vvA6P68XZ2QS/6Go1Gk0Q8DUMqDSgSkUVJ7douw40Etibtb/P6euIa4MWk/XRv3PdF5IJ+uL2hIe+U5Kfz9md/yc++9ktaP7yfT/1lB6vmP8XpX/sy907dyQPH/4KGmM2NPz6TdWd8l6/95CV2rnyHiSdcwD9+dwMj0v1c+PW55N74a77y9HLee+5NWqs3UTTlSE47+zBuPWkCaa89SDC/jAlHzeXrZ0/jnLHphJ/+DcseWcB76xuoCltk+Vw9f/LJ42ioWEK4qQbDF/D0/ClMm1bEGQeVcuSIbIraq7CWv0Pt+x+TWTyb/JFljBiTx7zJRcwqz2VCXho54Vpk20rC65dSv3oz9Wt20LCxkYlnTCV/ymjSx07EP2YKVu4IQmn51LZb7GiNUtkcZktDO5vr2jnG7/roZxS4Wn5GUQbBwiyCxfmunp+Xh5FXgplf3GshdMMXQMz4tp/WqOUVS+nQ80NRt4hKKGwl9HwrZrtJ1ZL88w1TEnp+MGAm9PyAz0zJPx/iCdecbvV8f9K2WxRd+pQUTTl2p0Lo0LOevyekqufvdRzAAGjlB7LnSkoI9MFjs1YpNbtfLityBTAbOD6pe6xSqlJEJgCvicgypdSGvbnOgD3pi0i6iHwoIktEZIWI/NjrHy8iH3hGjX+ISGCg5qDRaDR9Je6y2U+G3EpgdNL+KK+v8zVFTgF+AJynlIrE+5VSld5rBfAGcPge35jHQMo7EeAkpdRhwEzgDBE5GrgTuFspNQlowP05o9FoNIME8dJ5995SYCEw2XvYDQCXAZ28cETkcOBPuAv+zqT+fBFJ87aLgHlAsgF4jxiwRV+5tHq7fq8p4CTgKa//UeCCgZqDRqPR9JX+fNJXSlnA9cB8YBXwhFJqhYj8RETO8077HyALeLKLa+Z0YJGILAFeB+7o4vWzRwyopu+5K30ETMJ1W9oANHr/ELAbo4ZnELkWoDwjfSCnqdFoNAncNAz9Z9dQSr0AvNCl7/ak7VN6eN+7wCH9NhGPAfXeUUrZno/pKFzXpWl9eO/9SqnZSqnZmeOn8NVv/pqRR5zMyfOFj578G/Ouupp/n2zy15NuYG1rhOtuOp76q3/J5Xe8TtVH8xl7zLnc/415FARMLv3i4ZTf9jtu+s8a5j+9gOZtaymYcBgnnD2bH542mbz3/sbHv3qS8Ud/imvPmc5l0/Kw/nMvyx56nXeX17A1FCPLZ3BYbhrTThjLhAtPJNSwI8mIO42pM4o5/7ARHDM6l9JoNdayBdS+t5CqDyooGD2aEeNcI+4RI3OZVJBOXqwB2baSyNpPqF++kYY122moaKSmNkT+lDGkj3ONuHbuCCIZhdSFLHa2RdnaFGJLY4jNde1sq2+nIGCSlZ9ORlGQzNJMMkuyCRbnEyzMJVBYgJlfgplbiJFdgGNFd/l37mrENX0BDJ8fwxegNWLR1B7rZMRtCVtEkoKyrJiNYzmJylm+gIlhSiJAKzkoK+AzOypnpWjEBRJVs3oy4vrj1bO6+TT3VJELOoy48QpaiX8T7zX+JLc3ds39acTdk/GHvRHXQyS1NhTZJ947SqlGEXkdmAvkiYjPe9rv1qih0Wg0+xNjr7+SBy8D6b1TLCJ53nYQNyJtFa42dZF32lXAvwdqDhqNRtNXBP2kv6eUA496ur6Ba8B4XkRWAo+LyM9ww48fGsA5aDQaTZ8ZCvmM9pQBW/SVUkvpxqfU8zed05exKjbtYMxn57HsrrPInft15l5xJa9ckMPfZl/OkqYw37zxU7Tf+Fsu/NlrbHnvecbMPYc/fetTzF75OGVXz2T0L+/npvmbeeaxN2jctJy8cQdz/Llz+eXZ0ylZ9A8+/uVfefWj7Xzlf2dw1SFF2M/9jsX3zOfdxdVsao8RNIXDctM45PgxTL74RPzHXoThu4OssnEUT5rB5BnFXDBzJPPG5FEeq8FZvoDad96n6oMNVC+voez8PI6bWszcsflML8qg0GrAqFxJdO0n1C3dQN2qSurWNbBzZxs7whbBiZMJjJuGnT+aSGZxIihrS1OYLY0hKmra2FzbRnNjmKz8dDJLMl1N39PzM0rySSspcvX8/BKM3CKcYO4u/6670/MNf6CTnt8ajiX0/GjEwoo5OJbbrJhDeoa/Wz0/GDA76fkB0+iTnq8c20u4Jr3q+V316N3p+XGS9XxDetbz9+QnsdbzhyhD+Ck+FVL6LIvIhSKyTkSaRKRZRFpEpHmgJ6fRaDT7GulfP/1BR6pP+r8CzlVKrRrIyWg0Gs1gQMs7UK0XfI1GM1w4gNf8lBf9RSLyD+BfuOkVAFBK/XMgJqXRaDT7C10u0SUHaAdOS+pTwD5Z9H3BLFb99mxen3YUc7/xW17/dBZ/OcI14t5w03G0f+v3nP+TV9n87nOMmXsOD910HHNW/J1nv3QfF2x4lxs9I259xRIKJhzG8efO5X/Om+EacX/+KK98WEVV2OLmQ4txnvsdn/z+Bd76ZEfCiDsrL51DTxrH5EtPxn/cJay2chNG3BmHlHLBzJEcNzaPEVYNzrI3qHnrXSrfXU/18hrWtEQ5cXrJLkbcyMoPuzXi1kbthBE3nFlMjWfE3dQQ2sWI294cIbMks1NQVk9G3K6G3N0Zcc20IKYv0KsRNx6gZVtOykbcgM/okxEXSNmIm6yxaiOuZm84gNf81BZ9pdQXBnoiGo1GM1g4kAuNpOq9M0pEnhGRnV57WkQGtLqLRqPR7A/EK5eYShuKpPqF9mfcdKAjvPac16fRaDQHHDoiF4qVUsmL/CMicuMAzKdbDh6VzYvjZ7Ogtp3XzoQHj7iCta1RvnPbadR86U4+fftLVC58gQnHnc//3XQcB713H09d9xfeqQvx4nMV/Ocfr9K0ZRWFk2Zx+qeP4ednTqXgnUdY+PO/8+rianaELcZl+Ik99T98cs9LvLWsI8narLx0DjllHJMuPRXzuEtZFc7kiSVVlE4+iIMPLeXTh4/kU6NzKYtsx1ryOjVvf8C2d9ezY2Ut61tjVEcszh1XwNSiIIXROrdS1soPqV26gbqVVdSvr6emNkRlyKIhZtNqOVgFY4lkFFLTblHZ7CZZ21jvVspK1vPbWyKd9PzM8sKOJGuFZUhOEU56tqvpp2Un/j1T0fPjCde60/PjSdbier5tOynr+Wk+o096vlvhKjU9P67Fp6Lnw+4rZXXV82UP/8J70/P7+2FxiK5DgwpByzsAdSJyhYiYXrsCqBvIiWk0Gs3+QkRSakORVBf9LwKXADuA7bgJ07RxV6PRHHh4vwBTaUORVL13NgPn9XqiRqPRDHEEt4bDgcpuF30RuVkp9SsR+T2uX34nlFLfHLCZJVG/bA3vG+X88N7LuWvOtTRbNrfe/Rk+Of1mvnDLv6hevoDpp1/EP771Kcr+fQd//d4/+bgxzInFGXztL8/TWr2JkhnzuODCI/nJaZNI/+8feP8XT/PaqlpqIjYTMwMcN3ckC//3Bd5eV09V2CLXb3BkfpCDzprI+EvPwZh7IUuaTB77ZCtvLa5i1qxyLvCKphS3bSH2yWtUv/UhVe9vpGp1Hetbo1RHLEK2YkZxBnmhatiyjNCqjxN6ft36BnbWh9gRthN6ftRRtKcXUNtmUdkcYUtTmI11bQk9v7UxTFtzhFBrhEhLM1nluUl6fiFGXglmfrGr5wdzUcFc7LQs2mOuVp6s55v+gLftxwwEMfwBTF/A3fYFaGyPEorahMJWp6IpVtTGsRW256tvx4uoeFp+ctGUYMAkYMb33dYXPR/opOf7zQ79vquebxqp6/nQvZ7fX1p+8vhxtJ4/dBiq0k0q9CbvxFMvLMIte9i1aTQazQGFG5Hbf/KOiJwhImtEZL2I3NLN8TQR+Yd3/AMRGZd07Favf42InN4f97fbJ32l1HPeZrtS6skuE724Pyag0Wg0g43+es736oncg1tEahuwUESe7VLg/BqgQSk1SUQuA+4ELhWRGcBlwEG4rvKviMgUpVT3P11TJFVD7q0p9mk0Gs0Qx5ULU2kpMAdYr5SqUEpFgceB87uccz7wqLf9FHCyuPrS+cDjSqmIUmojsJ4+1iLpjt40/TOBs4CRIvK7pEM5gLW3F9doNJpBR98Cr4pEZFHS/v1KqfuT9kcCW5P2twFHdRkjcY5SyhKRJqDQ63+/y3tHpjyzHujNe6cKV88/j84afgvwrb29eKpEHcWPn7+FX/tPIMBTfP+JG3hsxAXccvP/0bxtLUdc/Dmeue5o7N98mwf+53U2tEU5d1QOJ93zJa788TJGHnkWX7j4EG7+1BjC//dTFtz5Iq9saaLVcjg4J415J45l+lcv4ucX3ElNxKY4zeSoggymXTid0Redj5pzAW9XtfP4x5v5cHEV1esq+MmlFzNnZDZ5dWuJfPQK2xcsovL9LWyraGR9a5TaqE3UUZgC+a1bcTYuoX3FYupWVFC7spqGikZ2NIbZEe4IyrI9U3l1u2vE3dQYYnNdOxU1rVTVhxJG3HB7lEhLM7H2JjJmFJJZVoi/MJ5krRiyChNJ1mx/Bm1Rh/aY0xGQZZiYfjcAK7lSls8z4MaDtFrDFtGo3a0R14ra2LYbnOXYDgHPgBvwGWQEzE5BWclG3IDP6GTE7TDadhhxkw2vjmOnbMTt7smrJyNunFSNuH01umoj7tBFlEJ6+dwkUauUmj2Q8+lvetP0lwBLRORvSin9ZK/RaIYFopz+GqoSGJ20P8rr6+6cbSLiA3Jxg19TeW+f2a2mLyJPeJufiMjSpLZMRJbu7cU1Go1m8KFAOam13lkITBaR8SISwDXMPtvlnGeBq7zti4DXlFLK67/M8+4ZD0wGPtzbu+tN3rnBez1nby+k0Wg0Qwa1S1jSHg6jLBG5HpgPmMDDSqkVIvITYJFS6lngIeD/RGQ9UI/7xYB33hPASlwb6nV767kDvcs7273NWiCklHJEZAowDXhxby+eKuUHjedzVYfxwp9+S+uH9/PddUU8dPN9OFaUM75yNf+4bDoV37icvz+2glbL4bNzRjD3dzezsPRYJh3/Ljd/biafHaPY+asbee/et1lQ2w7AvMIgR14wlYlfvprG6adRE/kFo4N+5ozNYdpnZlL+mYtpm3I8r1U08viirSxfWs3ODatp3bGJ48fmkr71I9ref5nKBYup+rCKjdua2RqyqE/S83P9JvbqD2hZvoS65RupW1NLQ0Ujla1RaiJuUFbI7tDzA4ZQUR9iS1OYTbVtVNS0Ut0Qoq05QntThPbWCLG2JqLtTVihVrJGFuMvKsXwiqaQmYeTnouTnkPMTKM9atMWc2iPqR71/OQka2aaq+v7An4iEQsrGi+WYnvBWG4BlWQ937asJC3f2K2eH0hOuJak53cNyHKSNFW/aWAIver5XSX9VPT83hKs7a323t3btZ4/yFEq1af4FIdTLwAvdOm7PWk7DHTrAq+U+jnw836bDKm7bC4A0kVkJPAS8Hngkf6ciEaj0QwWRDkptaFIqou+KKXagQuBe5VSF+MGDGg0Gs0BhgLHSq0NQVLNpy8iMhf4HG70GLj6lEaj0RxYKPpV3hlspLro34gbgfuMZ1yYALw+YLPqwqo6m6W//xNj5p7DyfOF9//+B7LKxvGtGy/klgmtvHvqWTz5YRUFAZMvXjydGb+6k7/X5HPH797l3uvmciwVrL31F7z+1GqWNIXJ9RscW5TJYV+cw8gvfIWKnOk88s5mpmenMfvQEqZeMof8cz/H9twp/HfFTh7/cCubVu6kvmI57XVVOFaUwMpXqX/nNSrfXsn2j3awvradqrBFU8zGVq42n+s3GJHup/6DD6hbvon69Q3UbW6iMuQWQG+Kudp/sp6f5TNYW9dGxc42Nte1UdcQor054vrnt4WJttQTC7dihVqxo2H8JRMxC8sw80sSxVJUMJcwPtqjDm0xh5Dl0BKxEgnVkn3zXf0+2FnP95KnxSJ2wh/f1fRVooBKXNNXjo1jRRN6fjDg61QwJa7jm4bsoulD73q+su1Oen5XX33o0PONJHW7Nz0//j5ITc/fk/xbA+2b3901NP2BAmeYL/pKqTeBN0UkS0SylFIVwD7JsKnRaDT7mqGq16dCqoXRDxGRT4AVwEoR+UhEtKav0WgOTPrPT3/Qkaq88yfg20qp1wFE5ATgAeCYgZmWRqPR7CeUgtTTMAw5Ul30M+MLPoBS6g0RyRygOWk0Gs1+5UCWd1Jd9CtE5Dbg/7z9K4CKgZnSroSaGpj3rc/z0jfmkjv364yZew73f/tY5q5+gmeOvpdXdrZxWG4653/nRPK/czffnb+eJ596herlCzj6rFo++MUjvPxeJVVhixHpPk44tITDrj2JjPOu5Z2WTO6fv4YPP9jGP04dx5TLTsZ//CWstgt4+qNKXly4jcq1VTRtWUWoYQcAadkFVD//LJXvrWPHkp2saXGrZLVa7gclaApFAR8jgz7KC4Ps+GAddesa2LmzLZFgrSnmVskCtzRb3Iib4zNZUdnM5to2mhvDtDdHaG+JEGlrJdbW1MmIa0VC+ErHYOQWJRKsOWnZtFuK9phNm+UQijk0hS2aItYuRtyEAdermuULpGGYBr6Aic9vYiUlW7M9422nwCwr6hpyY1HXgOsFZHU14iaaaWCI7LZKVtyIq+yk4CzD2G1AVtyAK5KaATf5er0Zcfe0gFIqRty9qc6kDbgDSf8GZw02+lIYvRj4J/A0UOT1aTQazYHHcNX0RSQd+CowCVgG3KSUiu2LiWk0Gs1+oZ/TMAw2epN3HgViwFvAmcB0XJ99jUajOSARhremP0MpdQiAiDxEP6T13BNGjCrj9dNivDztKI654Xf868tH0vCjr3DXve9TFY7x6ckFHP+H66g49GI++8cPWDr/TVqrN5E7ZjqvfOFuXt/RSsh2mJWXztwzJjDly5cRPfpi/r6qloffWMaGjzfQsHk5M/7nWuzDz+a1zc088ckGFi3eTvW6dTRXbcAKt2L4AqTnFpEzairrnruXrZsa2dgW61QwJctnUJrm6vklI7MpmFxA1YdVVDVH2BG2abY6F0wxBYKmQZbPIN9vUhAweKGqmdamMKGWKKHWCJGWxkSCNTsaxoqGcGJRHCuKFJRjB70Ea74g7VGHkKVoizm0RW2aIhZN4RitURszkL6rnp+UYM00DXx+E8Nn4PMbRCNWjwnW4lp+PDgrGDB7TLBmGkLANPAbgmHIbgumQGc9Xzl2r3p+QpdPUehO1vN3VzAlWXJPVQftjgNNz9+LqQ8RFNgHrvdOb5/lhJTT1yIqIjJaRF4XkZUiskJEbvD6C0TkZRFZ573m78G8NRqNZmCIp2E4QDX93hb9w0Sk2WstwKHxbRFp7uW9Fq4NYAZwNHCdV939FuBVpdRk4FVvX6PRaAYNB3KWzd7y6e9xUjUvF/92b7tFRFbhFvU9HzjBO+1R4A3ge3t6HY1Go+lfhrcht18QkXHA4cAHQGlScZYdQGkP77kWuBZgZG4Wd869jtqoxaunK9485gSeXr6T0jQf3/jiTCb+7Nc8vNnHXT9/jS0fvgzAmLnncPHZ03junHsoCJicMiafQ79wNKVXfIV1wQnc//IGXn5nM1XLF9NavQnl2GyfcjovLN7BE+9vYcvqGuorltJeV4VybHzpWWSWjKZgzGTKxuWx/Jl6toZiCX0+YAgFAZPSNB9jsgPkT8ijcGoR+VNG89b8ikSCtZDdUZGnwzffINfT83Oz02isaSPUGk0kWIu2N2FHQljhNmxPy4/r4XZ2MSotm5AyCSUlWGsMWbRGXf/81ohFc8RK0u+7T7Dm85v4AkZC229rjuzimx/X8ON6fkLT95u76PnxJGt+w8AUtxiK35BeE6wltr1+v2HsUvw8Wc83JDWduasPfyoJ1gaTlt/36/fvtQ58LT+JA3jR35vPdEqISBaub/+NSqlOkpBXB7LbumRKqfuVUrOVUrMLM4MDPU2NRqNxiadhSKUNQQZ00RcRP+6C/zel1D+97moRKfeOlwM7B3IOGo1G0zcUyoql1PaGVJxaRGSmiLznOcMsFZFLk449IiIbRWSx12amct0BW/TF/R37ELBKKXVX0qHkyu9XAf8eqDloNBpNn1Hsqyf9VJxa2oErlVIHAWcAvxGRvKTj31VKzfTa4lQuOpCa/jzcWrrLRCQ+me8DdwBPiMg1wGbgkgGcg0aj0fQJhepkWxpAenVqUUqtTdquEpGduClxGvf0ogO26Cul3qbnOJKT+zJWVVUTxXkFXPebi7lrzrVsaIty7qgcTvr9F6ic9yXO/McSFs9/m+Zta8kun8iME4/h/513ECfnNHFfThrzThzL9K9ehDrhSp5cU8ef/rWYDYs3U7f+Y2JtTfjSs8gdNYU7Xt/A+59UsWPtBpq3byDW1oQYJhmFI8gun0TJuDImTSzghGklrGqNJgKycv1GIsFaWXkWBZPzKZgygvzpY0kbP42toed6DMjK8RkUBEyK0nxkFAXJLM2kpSFEpKWZWHsT0bamXQKykg2SVrCAtphDe6JClk1L1KIpbNEatWmKxGgNWzS1x/CnZ7nBWZ4Rt7uArLhB1/QZXrWsngOyEoZcx05UzuopICtuzPWZRkoBWcn0FpDVtWpWd3SXiK0vAVl9NcDuTyNufxtwYbgZcelL5awiEVmUtH+/Uur+FN+bklNLHBGZAwSADUndPxeR2/F+KSilIr1ddJ9472g0Gs3QoU/59GuVUrN7OigirwBl3Rz6QacrKqVEpFunFm+cctwsx1cplXAtuhX3yyIA3I/7K+EnvU1YL/oajUaTjFJ7baTtGEqd0tMxEakWkXKl1PbdObWISA7wH+AHSqn3k8aO/0qIiMifge+kMqcBd9nUaDSaoYVKSJe9tb2kV6cWEQkAzwB/UUo91eVY3AtSgAuA5alcdEg86RfnpfOFVf/hVytsAjzFzTcew6jb7+KuxS08cNtLVH70MoYvwITjzufz503nuqNGkb7gURb/4Sku/dFZ5F9yLStlBH98fg0L3t3C9pWf0FazFYCs0nEUTjyECQeV8OJ/V9O4aRmhhmqUY+PPzCW7dBx5o8ZRPj6feVOLmTe+gINKMlniKIKmkO83KUv3MTo3jYLJBRRMKiR/+liyJk3CP246TuFYmmId+mDQFIJmh5ZfEDDJzk0jqySTjKIgWeU5tNdVd0qw1jUgK5mGsJ3Q8+PFUlo9Tb8t6mr5je0xWiMWZiDYKSDLFzBdTT8pICtZ209o+knFUpIDspykD38wSdM3PQ3fb7pJ0jp0fUnozb0FZCXv+80kDb+bgKxkjb8rvf1h9reW3x27GyPVJHGpogOy+oG4987A061Ti4jMBr6qlPqS13ccUCgiV3vvu9rz1PmbiBTj2k4X46bB75UhsehrNBrNvkP1xZC751dRqo5unFqUUouAL3nbfwX+2sP7T9qT6+pFX6PRaJJR7CuXzf2CXvQ1Go2mE33y3hlyDIlF3xo1nll3rWb9my/Q+uH9vGLO4OK7Pmbtm68QbWuieNrRzDv1EH585jQm133Mxlv+Hx89sZz360N852/PcvfS7Tz15odsXrKSpm1rsaMh0nOLyRt3MKOnjeSUw0dwzvRSjnv079jREGYgSEbhCHJGTaVsXD4zpxRx7MRCZo3IYVyOH3/1mo7kahk+CsfmUjS1kLwpo8ibOh7/uGlI+USsvFE02O4/ccAQgqaQaXZo+fmZfoJFGWSVZJBZmkmwJJ/MsgJCL+5IFD7vScsXw0QMk8Zwh19+XM9vibhafmvY3W4Nx2gJW/gzczv88BMF0A1Px++i7/sMrGjMvb7d2S9fOTZ2fN97Iopr+nEN3+8VQfebro7vNwTT0/RT8c1P3u+aXK1rH+yqjadiZOuq5+9Oy99T7b0nPV9r+YOYfvTeGYwMiUVfo9Fo9h36SV+j0WiGD/vOe2e/oBd9jUajSUKhEnWcD0T0oq/RaDTJ6Cf9/c+GTTvwv/4co448jZPnC0vn30dr9SZyx0xn9oXn8aNzZzAvrZrq+77Lfx98j3d2tlEftSlL9/HZhxdSsXgT9RuXEGtrwp+ZS/64gxkxbQLzZo7g3IPLmF2eSc7OlSjHThhwS8aWMGViAcdPLeGoUblMzE8j2LAJZ9ESmpcv5rDcNEpGZrsBWV5ytcC4aZgjp2Dnj6LZyKCmzWJbcztZPje5Wr5XHSs/4COzNIOMogwySzLIKMkls7yQYHE+/uJSok+v7za5WhwxTAxfADFMtrdGaPEqY7VEOwy4jaEYreEY7VGb1rBFNGoTSPN1CsBKBGf5TUyfYJgGgaQgKzsS6ja5WsKgaycFZ/nNbpOrxStmGSKJ7VQNuHHMpMpWycnVOhl26TBmphopmUpA1nAz4MIwN+KCa8iNRff3LAaMIbHoazQazb5j3wRn7S/0oq/RaDRd0fKORqPRDBOU6o9kaoOWIbHo+9Iz+d7PbuSW48eRO/frZJdPZM5ln+cH58/glLxW6v/yY1598B3e3tJETcSmOM3knPJspl04nf955t+JQimFk2ZRNmUiRx5WznmHlDN3VDZ59euIzH+ZjQsWUTztDIrGlDJlciEnTCthzsg8JuYHyGqpxPnkE9pWL6V26XpqV1Yz9djRnQqlmKNcLb/JzKImZFHZ3MamxhCb69oZke7bpVBKXMt3A7IK8RcWYeaXYOYXY4UW96rlm363GMr2loibYC0Uo6ndDcJqjVi0hGMJLd+K2Vgxh0DQv0uhFJ/f2EXLT/MZBAM+7GioVy0/Ps80n7FbLT8eqGX2oLv39EemHLtXLR86Cqz09Y81VS1/b2VureUPLbT3jkaj0QwXlELZetHXaDSaYYFSCidm7e9pDBh60ddoNJpkFPpJf39z8OgcvrnuId74ykscc8PvuP3cGRwXrGXnn3/EKw++y1vbW6mPulr+uaNymH7RwYy66AKcWeeiTvoeRVOOpHzKeOZ6fvlHjsgit3Y1kf++wsY3F1G1cBubNzRw7F03Jfzyx+elkdm0BeeTT2hd5Wr5dWtqqF9XT1VzhEt+cxlpE2ck/PIbjQxq2i22NbexpSlERU0bm+va2Fbbzk3Zabto+Qm//MIizMIyzPwSyMzHCeZ2m1ytq5Zv+PwYvgBbGtrdxGphi6ZQtJNffizi6vm27WBFbdIz/bv1y3eLm3v7prFLoZTutPy49pluGj365Rvi+trHC5wn39/utPw4cTvA7rR86HsZuPj5+1PL35PxB0LP13RGL/oajUYzTFBK4eh8+hqNRjN8OJC9d3RhdI1Go0nG895Jpe0NIlIgIi+LyDrvNb+H82wRWey1Z5P6x4vIByKyXkT+4RVR7xW96Gs0Gk0Sce+dVNpecgvwqlJqMvCqt98dIaXUTK+dl9R/J3C3UmoS0ABck8pFh4S8U790Nbd/s4GAIbx6umLjb67jaa8yVshWjMvwc8rUQqZdcgSlF15K4+g5/LOigcf/toTDz78gURnrkOJ0/Bs/oPWJV1jz5hKqPtpBRVULW0Mx6qM2t58+NVEZK/bORzSsWE7dio3Ura6joaKRre0xaiIWzZZD8KSLsfNGUWP7qGm32NzYwtamEBs9A+6OunbamiO0NUcom1nSqTJWsCQfX34xRn4JvsIynIw8nLRsnGAuUcP9so5XxhLDxPAHMDxjbtyAa6YFMX0BtjWEEpWxWsOWG4gVdbyALM+QaykcyyEzJ71TZaxgwCTNM+ImG3DjfVY0lEiO1p0Bt2PbTbgWr4zVUSWrswHXNe7uPilat0FpPSRW62rA7SnJWU/0ZMDtbpS+BlcNhAFXs+9w9o0h93zgBG/7UeAN4HupvFHcD+9JwGeT3v8j4I+9vVc/6Ws0Gk0ynstmivJOkYgsSmrX9uFKpUqp7d72DqC0h/PSvbHfF5ELvL5CoFEpFf+5sQ0YmcpFh8STvkaj0ewz+haRW6uUmt3TQRF5BSjr5tAPOl9SKRFRPQwzVilVKSITgNdEZBnQlOoEu6IXfY1Go0lC0X/eO0qpU3o6JiLVIlKulNouIuXAzh7GqPReK0TkDeBw4GkgT0R83tP+KKAylTkNiUU/6igunlXOzGtP5K4517KhLUrQFA7LTeewY0cz9fIT8Z9wGetUIX9esYMXnn2frasradqyisVP3Moopw5n+XPU/vkdKt9bx44lO1nfGqUqbNFquf9zg6Ywacf7RN/4iKpl66ldvpW6dQ3U1bRRGbJoiNk0xRyijvtlvDV9DNV1MTY1trKpvp2Kmja21bfT1BimrTlMqCVKqKWFWFsT5UdNIqMkn7SigkQglpFbhBPMxUrPxknPpd1StEcdQpblafcBxDQxk3R8wx/AFwi6mn4giOEPsLm2jUhSUjUradu2HGzbwfFe0zP9BDpp+R06fjzRWiCpObFoJ90+/oeQ3AfgODbpPi8gy9Py/YbRScdP1vVTTbYWx4xr971o+Xuru3d9e38nSdM6/hBBKZzoPknD8CxwFXCH9/rvrid4Hj3tSqmIiBQB84Bfeb8MXgcuAh7v6f3doTV9jUajSUaB4zgptb3kDuBUEVkHnOLtIyKzReRB75zpwCIRWQK8DtyhlFrpHfse8G0RWY+r8T+UykWHxJO+RqPR7CsU+ybLplKqDji5m/5FwJe87XeBQ3p4fwUwp6/X1Yu+RqPRJKPoVMf5QGNILPrlM8Yyfv7L3LO4igBPcdkR5Uy/ZDbFF36OncWH8PSGBh57ZgsbViyhdsMK2mq24lhRzECQvCd/zpq3l7H94x2s395GVdj1ybcVBAyhOM2kNM3HmAw/637zB2pX19GwqYnKkJXwyQ/ZDrZnVw8YQtAU/rOuloqdrk9+bUOI1sYw7a1Rwm1Roi31RNubsEKt2NEwhUcdkSiQ4mTk4aTnYqdnEzUCtMUc2tssQjFFUyRGS8TGH8xK+OSbaZ6Gn6Tjm4FgohBKc1OkW5/8eKI15Shsy8KxohRmBRI++UG/2UnHNw3ppOf7DTfhGuzqkw+ujh9H2TZpPqNbn/zk/eRCKMlj7Q63iMquSdW60/H3JA9Zqj75fY0B6O0amsGM0mkY9gQReVhEdorI8qS+lMKONRqNZr/RNz/9IcdAGnIfAc7o0pdq2LFGo9HsF5RS2FErpTYUGbBFXym1AKjv0n0+brgw3usFA3V9jUaj2TOUJ2n23oYi+1rTTzXsGC+c+VqAMeU9nqbRaDT9i66cNTD0EnaMUup+4H6AzJFT1FHXPkTTtrW0fng/jaPn8HJFA4+/sZU1K16lZv1K2nZuxY6GMANBMotHkzNqKqVj8njy+9cnEqrZyg30yfXHjbc+CsfmUjS1kLwpo/j3/77eo/E2yydkmgYFAZOCgMmf3t2cSKjWnfHWioRwLDe4yX/QFTjpucS8hGptMYf2sEMoFqMlatEUtmiKWLRGLVoiFoGs/ERCte6Mt6Zp4AuY+PwGrU2hhPHWtt2ALMd2EsZbZduJeRRkpXVKqNa1mV6ytHjlK8eKuf8vejDeJraTg7N6MN4mJ03rzYDb9Xg8OGt3xts9+cmabGA9kIy3urDWXqJA2T0uTUOefb3opxR2rNFoNPsLhdpXWTb3C/s6Ijcedgx9CBvWaDSafYYC5aiU2lBkwJ70ReQx3FzRRSKyDfghbpjxEyJyDbAZuGSgrq/RaDR7glJgR3VwVp9RSl3ew6Fdwo57I9TYQKClnpFHnMzJ84XNq16gcdMy2uuqXM08M5fsERMpGDORsnF5zJ1SzLwJhRxSkskvbwt7QVg+RqT7GJkVoGByPgWTCimYPpasyZMIjJuGUzSOJbe9mLhm0BSCpkGOzyDXb1KcZpKdm0ZGYZCs0kw2ragi1tbUSce3Y9GEfp6sS7fkT6QtpgiFHNpj0U4afmvEojli0dTuFUKJWATzyxJBWT6/iS8Q1/G9Aih+E8Nn4PMb7NzS1KHle9eOJ0pTjqvnO952SXZah37vBWP5DQO/KQk93zC8Vy8x2u50/GQy/GanhGjJOn6H7i496s270/lFpKOIStL7jS7n9JVdEq7tZoz+Tr5m9LPwrnX8fkQprelrNBrNcMLRi75Go9EME7TLpkaj0QwfFOAMUSNtKuhFX6PRaJJRShty9zdlI0v55wM3cFhpBrlzv44ZCBLML2XEEadTOiaPQ6cUceykIo4cmcO4HD/+6jXE1j5Py3+Xc3ppJoVjcymYlE/B9DHkTR2Pf9w0pHwidt4oGmwfNe0WmxvC5PqNTgFY+Zl+gkUZZJVkkFmaSbAkn4ziPDLKC2l4YEmnAKyuhkgxzERbXRemKewabpsibgBWU3uM1rC73Rr2jLhhCytmk1VU1CkAy0gKyjJ94hp3vQpYm1ds6xSAFW92fN/uyI5ZnJO2SwCW33SNtn4jXvWqY9uORRP301u1K79hdArASs6o2am/h/fvDtOz2O7OcLunhtaejLfacDt8UTo4S6PRaIYRetHXaDSa4YSOyNVoNJrhwz6KyE2lvoiInCgii5NaWEQu8I49IiIbk47NTOW6Q+JJvyRUQ9oNl/HaRzs45tu/7xR8Ve4LY25fRXT169Q/t5p1q7dSu7qOuqpWKkMW1/79m4ngKytvJDXtFjVtFpsa2tm8saP6VUNDmNvG5SWCrzJKssgoySejrIBgcQFGfgm+wjKMvGKcYC7h/72z0xyTNXzD71a6Mnx+DF+Ad7c0dAq+imv47Z6Gb8UcrKidqHaVW5iRCL6KJ1mLB1Wl+QyCAZ+7bxq831KfCL6Ka/jJVa7c5j61FKT7OwVfmV22DcHV/M2O4Kw43WnwyX0+s3PwlSEd+n1y0FZPY+0Og87a+y5BVX0aLel9uxlzl3P7OHZ/a/jJaD1/YFHsMz/9eH2RO0TkFm//e53motTrwExwvySA9cBLSad8Vyn1VF8uOiQWfY1Go9lnKIWzb7x3zsdNVQNufZE36LLod+Ei4EWlVPveXFTLOxqNRpOEUu6TfiptL0m5vojHZcBjXfp+LiJLReRuEUlL5aL6SV+j0Wi60IeqWEUisihp/36vFggAIvIKUNbN+37Q6Xq91BfxUtEfAsxP6r4V98sigFt75HvAT3qb8JBY9Cu3NfKnbWsJmsKrpysiq16g/vE11K3axobVddTsaGVH2KY2atFqOYSSvoGXH/pZNjaG2LymnYqaNWyubaOpMUxbc5hQS5RwW3sicdrsb55MsLQYM78YM78kod87wVyctGxaHaEtpmiPORi+QLf6veEP4Au4ydIMXwAzLcjrq3b2qN9bUbf4SXIRlOmzRhDwGWQETAI+M6HfxzX95MIn0bYmYFf9PlnXB7cASn7Q30m/9xtGothJd8VPetP0kwkY8QInnfX7+E/J7gqgpIqZ9Kaub98bf/qe3qsl82GO6tNTfK1SanbPQ6lTejomIn2pL3IJ8IxSKpY0dvxXQkRE/gx8J5UJa3lHo9FokvH89FNpe0lf6otcThdpx/uiQNwnqguA5alcdEg86Ws0Gs2+QrHPEq51W19ERGYDX1VKfcnbHweMBt7s8v6/iUgx7o/TxcBXU7moXvQ1Go0mGaWwowO/6Cul6uimvohSahHwpaT9TcDIbs47aU+uqxd9jUajSUIpcJROw7BfKc5J4+YvHkPB9HHcNedaGmI2rZZD1IuIM8U1JGb5DEak+ykIGBSn+cgoCPKle9+jvSVCpK3VNdi2NWGF23CsKFYklKguBZB9xR3Y6Tm0xhzaYg4hyyEUc2iqt2iKtNAa8SpeRSyyR0z0DLgBzEDQM+CmJVW1MhPJ0bZUNGB7hloraqOUSlS66lrlSjk2B4+c3qm6VaIlJUnzGwamgBVuAzobbKH7Klf5QX+3BtuuidFSCaLaNeFafIzOBtueKl31BaF7o+ueVMvqOm6q6IRpwwtbL/oajUYzPFDAAZxvTS/6Go1G0xX9pK/RaDTDBEeRkI4PRIbEou+MmcD7V/6KjfXtBHiKiZl+CgIm2YUZZBQFySzNJLMkm4yyQjJK8gkUFmAWlmPmF7PmS/9MaPbJJJKjeQFUpi/AUxujNEV2uNq9lyCtKRQjFLVoCVuEkgKsyqbOSGj28YInhuntewVO4sFUC+Yv7aTZd5cgLTmYalp5dkKz95nuq6vld2zHk6XF7RJd6a4vJ83XSbOPJ0jrWuAkrl/3JTGaz5Qei5zsbUESs8sA/V3gxB1Ta/aaDrS8o9FoNMMEhdLyjkaj0QwXtCFXo9Fohhl60d/PrNtczZe/8WucWJTWD++HzHw3CVp6DjFfkPaYQ8hSNMYcKqM2TRGLpnCM1qhNML90l0RoZpr76gv4XT3e862/+9mVXjK0zgnQHNvBtixXj/f86k8/d1bCd75rErSEj71p4DeEF/6ysVMitGStvDu/+skFmbtNhJZcrMSOhlL6N1SOTVbAVd27JkGD7v3q+0IgBd19T/3qkwuy9Cd90fG1Rj98UEp772g0Gs2wQaG9dzQajWbYoDV9jUajGWZoeUej0WiGCa6mv79nMXAMiUXfDKRTMmMeps/g5PmCFa3HitV4gVI2tqVwLCdRjUo5CtuycKwop332bM+4ahL0m52qT3VNaHbbDx9JCpJyuq0+FefyI87DEHYxtHZneA031Sbel0rA05hct9RlKtWn+hJAlel3R+rOJrm3AU9+s/MA/Wn3NAfIiqqNs5qe0E/6Go1GM0xQwD4pobKf0Iu+RqPRJKFQ2ntHo9Fohguu945e9PcrB48t4J3fnQNA7tyv9+m9j9x3ccrnfrtma8rnzhudnfK53SV82x0lmQPzvyXDv6dlTHrHNxBZ0Dy09q7ZpxzghtyBWwV2g4icISJrRGS9iNyyP+ag0Wg03RF/0k+l7Q0icrGIrBARxyuG3tN53a6XIjJeRD7w+v8hIoFUrrvPF30RMYF7gDOBGcDlIjJjX89Do9FoesJWqbW9ZDlwIbCgpxN6WS/vBO5WSk0CGoBrUrno/njSnwOsV0pVKKWiwOPA+fthHhqNRrMLDm4ahlTa3qCUWqWUWtPLad2ul+L6b58EPOWd9yhwQSrXFbWPDRYichFwhlLqS97+54GjlFLXdznvWuBab/dg3G/FA4UioLbXs4YOB9r9wIF3T8PpfsYqpYr3dGAR+a83fiqkA+Gk/fuVUvf38XpvAN9RSi3q5li36yXwI+B97ykfERkNvKiUOri36w1aQ673D3c/gIgsUkr1qHkNNfT9DH4OtHvS95M6Sqkz+mssEXkFKOvm0A+UUv/ur+v0hf2x6FcCo5P2R3l9Go1Gc0ChlDplL4foab2sA/JExKeUsujDOro/NP2FwGTP8hwALgOe3Q/z0Gg0msFOt+ulcnX514GLvPOuAlL65bDPF33vW+l6YD6wCnhCKbWil7f1SSMbAuj7GfwcaPek72eQISKfFpFtwFzgPyIy3+sfISIvQK/r5feAb4vIeqAQeCil6+5rQ65Go9Fo9h/7JThLo9FoNPsHvehrNBrNMGJQL/pDNV2DiDwsIjtFZHlSX4GIvCwi67zXfK9fROR33j0uFZFZ+2/m3SMio0XkdRFZ6YWN3+D1D8l7EpF0EflQRJZ49/Njr7/bsHYRSfP213vHx+3XG+gBETFF5BMRed7bH+r3s0lElonIYhFZ5PUNyc/cYGLQLvpDPF3DI0BXX99bgFeVUpOBV719cO9vsteuBf64j+bYFyzgJqXUDOBo4Drv/8VQvacIcJJS6jBgJnCGiBxNz2Ht1wANXv/d3nmDkRtwjX1xhvr9AJyolJqZ5JM/VD9zgwel1KBsuBbt+Un7twK37u959WH+44DlSftrgHJvuxxY423/Cbi8u/MGa8N1DTv1QLgnIAP4GDfKsRbwef2Jzx+u58Rcb9vnnSf7e+5d7mMU7iJ4EvA8bvGyIXs/3tw2AUVd+ob8Z25/t0H7pA+MBJJzHW/z+oYqpUqp7d72DqDU2x5S9+lJAYcDHzCE78mTQhYDO4GXgQ1Ao3Jd5KDznBP34x1vwnWRG0z8BriZjqJPhQzt+wE34eVLIvKRl5YFhvBnbrAwaNMwHMgopZSIDDlfWRHJAp4GblRKNUtSovuhdk9KKRuYKSJ5wDPAtP07oz1HRM4BdiqlPhKRE/bzdPqTTymlKkWkBHhZRFYnHxxqn7nBwmB+0j/Q0jVUi0g5gPe60+sfEvcpIn7cBf9vSql/et1D+p4AlFKNuJGNc/HC2r1DyXNO3I93PBc3DH6wMA84T0Q24WZhPAn4LUP3fgBQSlV6rztxv5jncAB85vY3g3nRP9DSNTyLGyoNnUOmnwWu9LwPjgaakn6+DgrEfaR/CFillLor6dCQvCcRKfae8BGRIK59YhU9h7Un3+dFwGvKE44HA0qpW5VSo5RS43D/Tl5TSn2OIXo/ACKSKSLZ8W3gNNxMu0PyMzeo2N9Ghd014CxgLa7e+oP9PZ8+zPsxYDsQw9UWr8HVTF8F1gGvAAXeuYLrpbQBWAbM3t/z7+Z+PoWrry4FFnvtrKF6T8ChwCfe/SwHbvf6JwAfAuuBJ4E0rz/d21/vHZ+wv+9hN/d2AvD8UL8fb+5LvLYi/vc/VD9zg6npNAwajUYzjBjM8o5Go9Fo+hm96Gs0Gs0wQi/6Go1GM4zQi75Go9EMI/Sir9FoNMMIvehr9jsiYnuZFFd4mS9vEpE9/myKyPeTtsdJUrZTjWa4oxd9zWAgpNxMigfhBkqdCfxwL8b7fu+naDTDE73oawYVyg25vxa43ouuNEXkf0RkoZcn/SsAInKCiCwQkf+IW3PhPhExROQOIOj9cvibN6wpIg94vyRe8qJwNZphiV70NYMOpVQFYAIluNHMTUqpI4EjgS+LyHjv1DnAN3DrLUwELlRK3ULHL4fPeedNBu7xfkk0Ap/ZZzej0Qwy9KKvGeychptTZTFuOudC3EUc4EOlVIVyM2Y+hpsuojs2KqUWe9sf4dY60GiGJTq1smbQISITABs3g6IA31BKze9yzgm4+YCS6SmnSCRp2wa0vKMZtugnfc2gQkSKgfuAPyg3MdR84GteamdEZIqXdRFgjpeF1QAuBd72+mPx8zUaTWf0k75mMBD05Bs/bj3e/wPiKZwfxJVjPvZSPNcAF3jHFgJ/ACbhphF+xuu/H1gqIh8DPxj46Ws0QwedZVMzJPHkne8opc7Zz1PRaIYUWt7RaDSaYYR+0tdoNJphhH7S12g0mmGEXvQ1Go1mGKEXfY1GoxlG6EVfo9FohhF60ddoNJphxP8Hu38ozVmDRBEAAAAASUVORK5CYII=\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "def draw_pos_encoding(pos_encoding):\n",
    "    plt.figure()\n",
    "    plt.pcolormesh(pos_encoding[0], cmap='RdBu') # 绘制分类图\n",
    "    plt.xlabel('Depth')\n",
    "    plt.xlim((0, 512))\n",
    "    plt.ylabel('Position')\n",
    "    plt.colorbar() # 条形bar颜色图例\n",
    "    plt.savefig(cwd+'imgs/pos_encoding.png')\n",
    "    plt.show()\n",
    "\n",
    "draw_pos_encoding(pos_encoding)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 3.掩码 masking\n",
    "这里用到的mask有2种：\n",
    "- padding mask：mask pad，即句子中为pad(value=0)的位置处其mask值为1\n",
    "- look-ahead mask：mask future token，将当前token后面的词mask掉，只让看到前面的词，即future token位置的mask值为1\n",
    "\n",
    "**【注意】：** 因为我这里使用的是torchtext里的tokenizer,从前面可以看出它的词表里pad的index=1，而不是常规的0。\n",
    "这里要特别注意，不然容易出错！"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 5])\n",
      "tensor([[7, 6, 0, 0, 1],\n",
      "        [1, 2, 3, 0, 0],\n",
      "        [0, 0, 0, 4, 5]])\n",
      "torch.Size([3, 1, 1, 5]) torch.float32\n",
      "tensor([[[[0., 0., 0., 0., 1.]]],\n",
      "\n",
      "\n",
      "        [[[1., 0., 0., 0., 0.]]],\n",
      "\n",
      "\n",
      "        [[[0., 0., 0., 0., 0.]]]])\n"
     ]
    }
   ],
   "source": [
    "pad = 1 # 重要！\n",
    "def create_padding_mask(seq):  # seq [b, seq_len]\n",
    "    # seq = torch.eq(seq, torch.tensor(0)).float() # pad=0的情况\n",
    "    seq = torch.eq(seq, torch.tensor(pad)).float()  # pad!=0\n",
    "    return seq[:, np.newaxis, np.newaxis, :]  # =>[b, 1, 1, seq_len]\n",
    "\n",
    "x = torch.tensor([[7, 6, 0, 0, 1],\n",
    "                  [1, 2, 3, 0, 0],\n",
    "                  [0, 0, 0, 4, 5]])\n",
    "print(x.shape) # [3,5]\n",
    "print(x)\n",
    "mask = create_padding_mask(x)\n",
    "print(mask.shape, mask.dtype) # [3,1,1,5]\n",
    "print(mask)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 3])\n",
      "tensor([[0.8287, 0.6121, 0.3410]])\n",
      "torch.Size([3, 3]) torch.float32\n",
      "tensor([[0., 1., 1.],\n",
      "        [0., 0., 1.],\n",
      "        [0., 0., 0.]])\n"
     ]
    }
   ],
   "source": [
    "# torch.triu(tensor, diagonal=0) 求上三角矩阵，diagonal默认为0表示主对角线的上三角矩阵\n",
    "# diagonal>0，则主对角上面的第|diagonal|条次对角线的上三角矩阵\n",
    "# diagonal<0，则主对角下面的第|diagonal|条次对角线的上三角矩阵\n",
    "def create_look_ahead_mask(size):  # seq_len\n",
    "    mask = torch.triu(torch.ones((size, size)), diagonal=1)\n",
    "    # mask = mask.device() #\n",
    "    return mask  # [seq_len, seq_len]\n",
    "\n",
    "x = torch.rand(1,3)\n",
    "print(x.shape)\n",
    "print(x)\n",
    "mask = create_look_ahead_mask(x.shape[1])\n",
    "print(mask.shape, mask.dtype)\n",
    "print(mask)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 4.scaled dot product attention\n",
    "![jupyter-img2](./imgs/im2.jpg)\n",
    "\n",
    "$$Attention(Q,K,V)=softmax_{(k)}(\\frac{QK^T}{\\sqrt{d_k}})V$$\n",
    "\n",
    "注意：实现时对mask的处理\n",
    "\n",
    "mask=1的位置是pad或者future token，乘以-1e9（-1*10^9）成为负无穷，经过softmax后会趋于0"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "source": [
    "def scaled_dot_product_attention(q, k, v, mask=None):\n",
    "    \"\"\"\n",
    "    #计算注意力权重。\n",
    "    q, k, v 必须具有匹配的前置维度。 且dq=dk\n",
    "    k, v 必须有匹配的倒数第二个维度，例如：seq_len_k = seq_len_v。\n",
    "    #虽然 mask 根据其类型（填充或前瞻）有不同的形状，\n",
    "    #但是 mask 必须能进行广播转换以便求和。\n",
    "\n",
    "    #参数:\n",
    "        q: 请求的形状 == (..., seq_len_q, depth)\n",
    "        k: 主键的形状 == (..., seq_len_k, depth)\n",
    "        v: 数值的形状 == (..., seq_len_v, depth_v)  seq_len_k = seq_len_v\n",
    "        mask: Float 张量，其形状能转换成\n",
    "              (..., seq_len_q, seq_len_k)。默认为None。\n",
    "\n",
    "    #返回值:\n",
    "        #输出，注意力权重\n",
    "    \"\"\"\n",
    "    # matmul(a,b)矩阵乘:a b的最后2个维度要能做乘法，即a的最后一个维度值==b的倒数第2个纬度值，\n",
    "    # 除此之外，其他维度值必须相等或为1（为1时会广播）\n",
    "    matmul_qk = torch.matmul(q, k.transpose(-1, -2))  # 矩阵乘 =>[..., seq_len_q, seq_len_k]\n",
    "\n",
    "    # 缩放matmul_qk\n",
    "    dk = torch.tensor(k.shape[-1], dtype=torch.float32)  # k的深度dk，或叫做depth_k\n",
    "    scaled_attention_logits = matmul_qk / torch.sqrt(dk)  # [..., seq_len_q, seq_len_k]\n",
    "\n",
    "    # 将 mask 加入到缩放的张量上(重要！)\n",
    "    if mask is not None:  # mask: [b, 1, 1, seq_len]\n",
    "        # mask=1的位置是pad，乘以-1e9（-1*10^9）成为负无穷，经过softmax后会趋于0\n",
    "        scaled_attention_logits += (mask * -1e9)\n",
    "\n",
    "    # softmax 在最后一个轴（seq_len_k）上归一化\n",
    "    attention_weights = torch.nn.functional.softmax(scaled_attention_logits, dim=-1)  # [..., seq_len_q, seq_len_k]\n",
    "\n",
    "    output = torch.matmul(attention_weights, v)  # =>[..., seq_len_q, depth_v]\n",
    "    return output, attention_weights  # [..., seq_len_q, depth_v], [..., seq_len_q, seq_len_k]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "execution_count": 15,
   "outputs": []
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "outputs": [],
   "source": [
    "def print_out(q, k, v):\n",
    "    temp_out, temp_attn = scaled_dot_product_attention(q, k, v, None)\n",
    "    print('Attention weights are:')\n",
    "    print(temp_attn)\n",
    "    print('Output is:')\n",
    "    print(temp_out)\n",
    "\n",
    "np.set_printoptions(suppress=True) # 设置不以科学计数法的形式显示数据\n",
    "\n",
    "temp_k = torch.tensor([[10,0,0],\n",
    "                       [0,10,0],\n",
    "                       [0,0,10],\n",
    "                       [0,0,10]], dtype=torch.float32) # [4,3]\n",
    "temp_v = torch.tensor([[1,0],\n",
    "                       [10,0],\n",
    "                       [100,5],\n",
    "                       [1000,6]], dtype=torch.float32) #[4,2]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tensor([[8.4333e-26, 1.0000e+00, 8.4333e-26, 8.4333e-26]])\n",
      "Output is:\n",
      "tensor([[1.0000e+01, 9.2766e-25]])\n"
     ]
    }
   ],
   "source": [
    "# query aligns with第2个key (key的第2列)，得到attention weights：[0. 1. 0. 0.]\n",
    "# 所以第2行 的value值被返回\n",
    "temp_q = torch.tensor([[0,10,0]], dtype=torch.float32) # [1,3]\n",
    "print_out(temp_q, temp_k, temp_v)\n",
    "# Attention weights：[0. 1. 0. 0.]， Output：[[10.  0.]]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tensor([[4.2166e-26, 4.2166e-26, 5.0000e-01, 5.0000e-01]])\n",
      "Output is:\n",
      "tensor([[550.0000,   5.5000]])\n"
     ]
    }
   ],
   "source": [
    "# query aligns with 重复的key (key的第3列和第4列)，得到attention weights：[0. 0. 0.5 0.5]\n",
    "# 所以第3行的value值与第4行的value值平均化后，被返回 【(100+1000)/2=550, (5+6)/2=5.5】\n",
    "temp_q = torch.tensor([[0,0,10]], dtype=torch.float32) # [1,3]\n",
    "print_out(temp_q, temp_k, temp_v)\n",
    "# Attention weights：[0. 0. 0.5 0.5]， Output：[[550.  5.5]]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tensor([[5.0000e-01, 5.0000e-01, 4.2166e-26, 4.2166e-26]])\n",
      "Output is:\n",
      "tensor([[5.5000e+00, 4.6383e-25]])\n"
     ]
    }
   ],
   "source": [
    "# query aligns with 第1个key (key的第1列)和第2个key (key的第2列)，得到attention weights：[0.5 0.5 0. 0.]\n",
    "# 所以第1行的value值与第2行的value值平均化后，被返回 【(1+10)/2=5.5, (0.+0.)/2=0.】\n",
    "temp_q = torch.tensor([[10,10,0]], dtype=torch.float32) # [1,3]\n",
    "print_out(temp_q, temp_k, temp_v)\n",
    "# Attention weights：[0.5 0.5 0. 0.]， Output：[[5.5.  0.]]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tensor([[4.2166e-26, 4.2166e-26, 5.0000e-01, 5.0000e-01],\n",
      "        [8.4333e-26, 1.0000e+00, 8.4333e-26, 8.4333e-26],\n",
      "        [5.0000e-01, 5.0000e-01, 4.2166e-26, 4.2166e-26]])\n",
      "Output is:\n",
      "tensor([[5.5000e+02, 5.5000e+00],\n",
      "        [1.0000e+01, 9.2766e-25],\n",
      "        [5.5000e+00, 4.6383e-25]])\n"
     ]
    },
    {
     "data": {
      "text/plain": "'\\n# Attention weights：tensor(\\n[[0.  0.  0.5 0.5]\\n [0.  1.  0.  0. ]\\n [0.5 0.5 0.  0. ]], shape=(3, 4), dtype=float32)，\\n# Output：tensor(\\n[[550.    5.5]\\n [ 10.    0. ]\\n [  5.5   0. ]], shape=(3, 2), dtype=float32)\\n'"
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 传入所有的query\n",
    "temp_q = torch.tensor([[0, 0, 10], [0, 10, 0], [10, 10, 0]], dtype=torch.float32)  # (3, 3)\n",
    "print_out(temp_q, temp_k, temp_v)\n",
    "\"\"\"\n",
    "# Attention weights：tensor(\n",
    "[[0.  0.  0.5 0.5]\n",
    " [0.  1.  0.  0. ]\n",
    " [0.5 0.5 0.  0. ]], shape=(3, 4), dtype=float32)，\n",
    "# Output：tensor(\n",
    "[[550.    5.5]\n",
    " [ 10.    0. ]\n",
    " [  5.5   0. ]], shape=(3, 2), dtype=float32)\n",
    "\"\"\""
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 5.multi-head attention\n",
    "![jupyter-img3](./imgs/im3.jpg)\n",
    "\n",
    "多头注意允许模型在不同的位置联合处理来自不同表示子空间的信息。\n",
    "（Multi-head attention allows the model to jointly attend to information\n",
    "from different representation subspaces at different positions.）\n",
    "\n",
    "$$\\begin{array}{ll}  & MultiHead(Q,K,V)=Concat(head_1,...,head_h)W^O \\\\ & where\\; head_i=Attention(QW_i^Q,KW_i^K,VW_i^V)\\end{array}$$\n",
    "\n",
    "其中投影维参数矩阵$W_i^Q\\in R^{d_{model}\\text{x}\\,d_k}$，$W_i^K\\in R^{d_{model}\\text{x}\\,d_k}$， $W_i^V\\in R^{d_{model}\\text{x}\\,d_v}$，$W^O\\in R^{hd_v\\text{x}\\,d_{model}}$。\n",
    "\n",
    "$h=8,\\;d_k = d_v = d_{model}/h = 64$\n",
    "\n",
    "multi-head attention有4部分组成：\n",
    "- linear layer 将 linear layer的结果 split到不同的head\n",
    "- scaled dot-product attention\n",
    "- 将所有的head 进行拼接 concat\n",
    "- final linear layer"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 60, 512])\n",
      "torch.Size([1, 60, 512]) torch.Size([1, 8, 60, 60])\n"
     ]
    }
   ],
   "source": [
    "class MultiHeadAttention(torch.nn.Module):\n",
    "    def __init__(self, d_model, num_heads):\n",
    "        super(MultiHeadAttention, self).__init__()\n",
    "        self.num_heads = num_heads\n",
    "        self.d_model = d_model\n",
    "\n",
    "        assert d_model % self.num_heads == 0  # 因为输入要被（平均？）split到不同的head\n",
    "\n",
    "        self.depth = d_model // self.num_heads  # 512/8=64，所以在scaled dot-product atten中dq=dk=64,dv也是64\n",
    "\n",
    "        self.wq = torch.nn.Linear(d_model, d_model)\n",
    "        self.wk = torch.nn.Linear(d_model, d_model)\n",
    "        self.wv = torch.nn.Linear(d_model, d_model)\n",
    "\n",
    "        self.final_linear = torch.nn.Linear(d_model, d_model)\n",
    "\n",
    "    def split_heads(self, x, batch_size):  # x [b, seq_len, d_model]\n",
    "        x = x.view(batch_size, -1, self.num_heads,\n",
    "                   self.depth)  # [b, seq_len, d_model=512]=>[b, seq_len, num_head=8, depth=64]\n",
    "        return x.transpose(1, 2)  # [b, seq_len, num_head=8, depth=64]=>[b, num_head=8, seq_len, depth=64]\n",
    "\n",
    "    def forward(self, q, k, v, mask):  # q=k=v=x [b, seq_len, embedding_dim] embedding_dim其实也=d_model\n",
    "        batch_size = q.shape[0]\n",
    "\n",
    "        q = self.wq(q)  # =>[b, seq_len, d_model]\n",
    "        k = self.wk(k)  # =>[b, seq_len, d_model]\n",
    "        v = self.wv(v)  # =>[b, seq_len, d_model]\n",
    "\n",
    "        q = self.split_heads(q, batch_size)  # =>[b, num_head=8, seq_len, depth=64]\n",
    "        k = self.split_heads(k, batch_size)  # =>[b, num_head=8, seq_len, depth=64]\n",
    "        v = self.split_heads(v, batch_size)  # =>[b, num_head=8, seq_len, depth=64]\n",
    "\n",
    "        scaled_attention, attention_weights = scaled_dot_product_attention(q, k, v, mask)\n",
    "        # => [b, num_head=8, seq_len_q, depth=64], [b, num_head=8, seq_len_q, seq_len_k]\n",
    "\n",
    "        scaled_attention = scaled_attention.transpose(1, 2)  # =>[b, seq_len_q, num_head=8, depth=64]\n",
    "        # 转置操作让张量存储结构扭曲，直接使用view方法会失败，可以使用reshape方法\n",
    "        concat_attention = scaled_attention.reshape(batch_size, -1, self.d_model)  # =>[b, seq_len_q, d_model=512]\n",
    "\n",
    "        output = self.final_linear(concat_attention)  # =>[b, seq_len_q, d_model=512]\n",
    "        return output, attention_weights  # [b, seq_len_q, d_model=512], [b, num_head=8, seq_len_q, seq_len_k]\n",
    "\n",
    "\n",
    "temp_mha = MultiHeadAttention(d_model=512, num_heads=8)\n",
    "x = torch.rand(1, 60, 512) # [b,seq_len,d_model,embedding_dim]\n",
    "print(x.shape)\n",
    "out, attn_weights = temp_mha(x, x, x, mask=None)\n",
    "print(out.shape, attn_weights.shape) # [1, 60, 512], [1, 8, 60, 60]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 6.point wise feed forward network\n",
    "2层线性变换和一个ReLU激活：\n",
    "$$FFN(x) = max(0, xW1 + b1)W2 + b2$$\n",
    "\n",
    "这一层的input和output的维度都是$d_{model}$，而内层的维度的是$d_{ff}=2048$\n",
    "\n",
    "其实就是(nn.relu(x w1+b1))w2+b2"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 50, 512])\n"
     ]
    }
   ],
   "source": [
    "# 点式前馈网络\n",
    "def point_wise_feed_forward_network(d_model, dff):\n",
    "    feed_forward_net = torch.nn.Sequential(\n",
    "        torch.nn.Linear(d_model, dff),  # [b, seq_len, d_model]=>[b, seq_len, dff=2048]\n",
    "        torch.nn.ReLU(),\n",
    "        torch.nn.Linear(dff, d_model),  # [b, seq_len, dff=2048]=>[b, seq_len, d_model=512]\n",
    "    )\n",
    "    return feed_forward_net\n",
    "\n",
    "sample_ffn = point_wise_feed_forward_network(512, 2048)\n",
    "input = torch.rand(64, 50, 512) # [b, seq_len, d_model]\n",
    "print(sample_ffn(input).shape) # [b=64, seq_len=50, d_model=512]\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "transformer model:\n",
    "\n",
    "(1) input sentence 传入N个encoder layer，句子的每个token都会有一个输出\n",
    "\n",
    "(2) decoder attends on encoder 的输出（encoder-decoder attention） 和 它自己的输入（self-attention），然后预测出下一个词\n",
    "\n",
    "## 7.encoder layer\n",
    "每个编码器层包括以下2个子层：\n",
    "- 多头注意力（有padding mask）\n",
    "- 点式前馈网络（Point wise feed forward networks）\n",
    "\n",
    "注意：每个子层还伴随着一个残差连接，然后进行“层归一化”（LayerNorm）。残差连接有助于避免深度网络中的梯度消失问题。\n",
    "\n",
    "每个子层的输出是 LayerNorm(x + Sublayer(x))。归一化是在 d_model（最后一个）维度完成的。\n",
    "\n",
    "注意：实现时在每个sub layer 之后加入了dropout层，再才进行add the sub layer input 和 normalized，即\n",
    "LayerNorm(x + Dropout(Sublayer(x)))\n",
    "\n",
    "Transformer 中有 N 个编码器层。"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "source": [
    "class EncoderLayer(torch.nn.Module):\n",
    "    def __init__(self, d_model, num_heads, dff, rate=0.1):\n",
    "        super(EncoderLayer, self).__init__()\n",
    "\n",
    "        self.mha = MultiHeadAttention(d_model, num_heads)  # 多头注意力（padding mask）(self-attention)\n",
    "        self.ffn = point_wise_feed_forward_network(d_model, dff)\n",
    "\n",
    "        self.layernorm1 = torch.nn.LayerNorm(normalized_shape=d_model, eps=1e-6)\n",
    "        self.layernorm2 = torch.nn.LayerNorm(normalized_shape=d_model, eps=1e-6)\n",
    "\n",
    "        self.dropout1 = torch.nn.Dropout(rate)\n",
    "        self.dropout2 = torch.nn.Dropout(rate)\n",
    "\n",
    "    # x [b, inp_seq_len, embedding_dim] embedding_dim其实也=d_model\n",
    "    # mask [b,1,1,inp_seq_len]\n",
    "    def forward(self, x, mask):\n",
    "        attn_output, _ = self.mha(x, x, x, mask)  # =>[b, seq_len, d_model]\n",
    "        attn_output = self.dropout1(attn_output)\n",
    "        out1 = self.layernorm1(x + attn_output)  # 残差&层归一化 =>[b, seq_len, d_model]\n",
    "\n",
    "        ffn_output = self.ffn(out1)  # =>[b, seq_len, d_model]\n",
    "        ffn_output = self.dropout2(ffn_output)\n",
    "        out2 = self.layernorm2(out1 + ffn_output)  # 残差&层归一化 =>[b, seq_len, d_model]\n",
    "\n",
    "        return out2  # [b, seq_len, d_model]\n",
    "\n",
    "# layernorm = torch.nn.LayerNorm(normalized_shape=512, eps=1e-6)\n",
    "# x = torch.rand(4, 50, 512)\n",
    "# print(layernorm(x).shape)\n",
    "\n",
    "sample_encoder_layer = EncoderLayer(512, 8, 2048)\n",
    "x = torch.rand(64, 50, 512) # [b, seq_len, d_model]\n",
    "sample_encoder_layer_output = sample_encoder_layer(x, None)\n",
    "print(sample_encoder_layer_output.shape) # [64, 50, 512]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "execution_count": 18,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 50, 512])\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 8.decoder layer\n",
    "\n",
    "每个解码器层包括以下3个子层：\n",
    "- masked的多头注意力（look ahead mask 和 padding mask）(self-attention)\n",
    "- 多头注意力（padding mask）(encoder-decoder attention)。\n",
    "    V和 K接收encoder的输出作为输入。Q接收masked的多头注意力子层的输出。\n",
    "- 点式前馈网络\n",
    "\n",
    "每个子层还伴随着一个残差连接，然后进行“层归一化”（LayerNorm）。\n",
    "\n",
    "每个子层的输出是 LayerNorm(x + Sublayer(x))。归一化是在 d_model（最后一个）维度完成的。\n",
    "\n",
    "Transformer 中共有 N 个解码器层。\n",
    "\n",
    "当 Q 接收到decoder的第一个注意力块的输出，并且 K 接收到encoder的输出时，注意力权重表示根据encoder的输出赋予decoder输入的重要性。\n",
    "换一种说法，decoder通过查看encoder输出和对其自身输出的自注意力，预测下一个词。"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 40, 512])\n"
     ]
    }
   ],
   "source": [
    "class DecoderLayer(torch.nn.Module):\n",
    "    def __init__(self, d_model, num_heads, dff, rate=0.1):\n",
    "        super(DecoderLayer, self).__init__()\n",
    "\n",
    "        self.mha1 = MultiHeadAttention(d_model,\n",
    "                                       num_heads)  # masked的多头注意力（look ahead mask 和 padding mask）(self-attention)\n",
    "        self.mha2 = MultiHeadAttention(d_model, num_heads)  # 多头注意力（padding mask）(encoder-decoder attention)\n",
    "\n",
    "        self.ffn = point_wise_feed_forward_network(d_model, dff)\n",
    "\n",
    "        self.layernorm1 = torch.nn.LayerNorm(normalized_shape=d_model, eps=1e-6)\n",
    "        self.layernorm2 = torch.nn.LayerNorm(normalized_shape=d_model, eps=1e-6)\n",
    "        self.layernorm3 = torch.nn.LayerNorm(normalized_shape=d_model, eps=1e-6)\n",
    "\n",
    "        self.dropout1 = torch.nn.Dropout(rate)\n",
    "        self.dropout2 = torch.nn.Dropout(rate)\n",
    "        self.dropout3 = torch.nn.Dropout(rate)\n",
    "\n",
    "    # x [b, targ_seq_len, embedding_dim] embedding_dim其实也=d_model=512\n",
    "    # look_ahead_mask [b, 1, targ_seq_len, targ_seq_len] 这里传入的look_ahead_mask应该是已经结合了look_ahead_mask和padding mask的mask\n",
    "    # enc_output [b, inp_seq_len, d_model]\n",
    "    # padding_mask [b, 1, 1, inp_seq_len]\n",
    "    def forward(self, x, enc_output, look_ahead_mask, padding_mask):\n",
    "        attn1, attn_weights_block1 = self.mha1(x, x, x,\n",
    "                                               look_ahead_mask)  # =>[b, targ_seq_len, d_model], [b, num_heads, targ_seq_len, targ_seq_len]\n",
    "        attn1 = self.dropout1(attn1)\n",
    "        out1 = self.layernorm1(x + attn1)  # 残差&层归一化 [b, targ_seq_len, d_model]\n",
    "\n",
    "        # Q: receives the output from decoder's first attention block，即 masked multi-head attention sublayer\n",
    "        # K V: V (value) and K (key) receive the encoder output as inputs\n",
    "        attn2, attn_weights_block2 = self.mha2(out1, enc_output, enc_output,\n",
    "                                               padding_mask)  # =>[b, targ_seq_len, d_model], [b, num_heads, targ_seq_len, inp_seq_len]\n",
    "        attn2 = self.dropout2(attn2)\n",
    "        out2 = self.layernorm2(out1 + attn2)  # 残差&层归一化 [b, targ_seq_len, d_model]\n",
    "\n",
    "        ffn_output = self.ffn(out2)  # =>[b, targ_seq_len, d_model]\n",
    "        ffn_output = self.dropout3(ffn_output)\n",
    "        out3 = self.layernorm3(out2 + ffn_output)  # 残差&层归一化 =>[b, targ_seq_len, d_model]\n",
    "\n",
    "        return out3, attn_weights_block1, attn_weights_block2\n",
    "        # [b, targ_seq_len, d_model], [b, num_heads, targ_seq_len, targ_seq_len], [b, num_heads, targ_seq_len, inp_seq_len]\n",
    "\n",
    "\n",
    "sample_decoder_layer = DecoderLayer(512, 8, 2048) #\n",
    "y = torch.rand(64, 40, 512) # [b, seq_len, d_model]\n",
    "sample_decoder_layer_output,_,_ = sample_decoder_layer(y, sample_encoder_layer_output, None, None)\n",
    "print(sample_decoder_layer_output.shape) # [64, 40, 512]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 9.encoder\n",
    "编码器 包括：\n",
    "- 输入嵌入（Input Embedding）\n",
    "- 位置编码（Positional Encoding）\n",
    "- N 个编码器层（encoder layers）\n",
    "\n",
    "输入经过嵌入（embedding）后，该嵌入与位置编码相加。该加法结果的输出是编码器层的输入。编码器的输出是解码器的输入\n",
    "\n",
    "注意：缩放 embedding\n",
    "\n",
    "原始论文的3.4节Embeddings and Softmax最后一句有提到： In the embedding layers, we multiply those weights by $\\sqrt{d_{model}}$."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 42, 512])\n"
     ]
    }
   ],
   "source": [
    "class Encoder(torch.nn.Module):\n",
    "    def __init__(self,\n",
    "                 num_layers,  # N个encoder layer\n",
    "                 d_model,\n",
    "                 num_heads,\n",
    "                 dff,  # 点式前馈网络内层fn的维度\n",
    "                 input_vocab_size,  # 输入词表大小（源语言（法语））\n",
    "                 maximun_position_encoding,\n",
    "                 rate=0.1):\n",
    "        super(Encoder, self).__init__()\n",
    "\n",
    "        self.num_layers = num_layers\n",
    "        self.d_model = d_model\n",
    "\n",
    "        self.embedding = torch.nn.Embedding(num_embeddings=input_vocab_size, embedding_dim=d_model)\n",
    "        self.pos_encoding = positional_encoding(maximun_position_encoding,\n",
    "                                                d_model)  # =>[1, max_pos_encoding, d_model=512]\n",
    "\n",
    "        # self.enc_layers = [EncoderLayer(d_model, num_heads, dff, rate).cuda() for _ in range(num_layers)] # 不行\n",
    "        self.enc_layers = torch.nn.ModuleList([EncoderLayer(d_model, num_heads, dff, rate) for _ in range(num_layers)])\n",
    "\n",
    "        self.dropout = torch.nn.Dropout(rate)\n",
    "\n",
    "    # x [b, inp_seq_len]\n",
    "    # mask [b, 1, 1, inp_sel_len]\n",
    "    def forward(self, x, mask):\n",
    "        inp_seq_len = x.shape[-1]\n",
    "\n",
    "        # adding embedding and position encoding\n",
    "        x = self.embedding(x)  # [b, inp_seq_len]=>[b, inp_seq_len, d_model]\n",
    "        # 缩放 embedding 原始论文的3.4节有提到： In the embedding layers, we multiply those weights by \\sqrt{d_model}.\n",
    "        x *= torch.sqrt(torch.tensor(self.d_model, dtype=torch.float32))\n",
    "        pos_encoding = self.pos_encoding[:, :inp_seq_len, :]\n",
    "        pos_encoding = pos_encoding.cuda()  # ###############\n",
    "        x += pos_encoding  # [b, inp_seq_len, d_model]\n",
    "\n",
    "        x = self.dropout(x)\n",
    "\n",
    "        for i in range(self.num_layers):\n",
    "            x = self.enc_layers[i](x, mask)  # [b, inp_seq_len, d_model]=>[b, inp_seq_len, d_model]\n",
    "        return x  # [b, inp_seq_len, d_model]\n",
    "\n",
    "\n",
    "sample_encoder = Encoder(num_layers=2,\n",
    "                         d_model=512,\n",
    "                         num_heads=8,\n",
    "                         dff=2048,\n",
    "                         input_vocab_size=8500,\n",
    "                         maximun_position_encoding=10000)\n",
    "sample_encoder = sample_encoder.to(device)\n",
    "\n",
    "x = torch.tensor(np.random.randint(low=0, high=200, size=(64, 42))) # [b, seq_len]\n",
    "# print(x.shape)\n",
    "sample_encoder_output = sample_encoder(x.cuda(), None)\n",
    "print(sample_encoder_output.shape) # [b, seq_len, d_model]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 10.decoder\n",
    "解码器包括：\n",
    "- 输出嵌入（Output Embedding）\n",
    "- 位置编码（Positional Encoding）\n",
    "\n",
    "N 个解码器层（decoder layers）\n",
    "\n",
    "目标（target）经过一个嵌入后，该嵌入和位置编码相加。该加法结果是解码器层的输入。解码器的输出是最后的线性层的输入。"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 36, 512])\n",
      "torch.Size([64, 8, 36, 36])\n",
      "torch.Size([64, 8, 36, 42])\n"
     ]
    }
   ],
   "source": [
    "class Decoder(torch.nn.Module):\n",
    "    def __init__(self,\n",
    "                 num_layers,  # N个encoder layer\n",
    "                 d_model,\n",
    "                 num_heads,\n",
    "                 dff,  # 点式前馈网络内层fn的维度\n",
    "                 target_vocab_size,  # target词表大小（目标语言（英语））\n",
    "                 maximun_position_encoding,\n",
    "                 rate=0.1):\n",
    "        super(Decoder, self).__init__()\n",
    "\n",
    "        self.num_layers = num_layers\n",
    "        self.d_model = d_model\n",
    "\n",
    "        self.embedding = torch.nn.Embedding(num_embeddings=target_vocab_size, embedding_dim=d_model)\n",
    "        self.pos_encoding = positional_encoding(maximun_position_encoding,\n",
    "                                                d_model)  # =>[1, max_pos_encoding, d_model=512]\n",
    "\n",
    "        # self.dec_layers = [DecoderLayer(d_model, num_heads, dff, rate).cuda() for _ in range(num_layers)] # 不行\n",
    "        self.dec_layers = torch.nn.ModuleList([DecoderLayer(d_model, num_heads, dff, rate) for _ in range(num_layers)])\n",
    "\n",
    "        self.dropout = torch.nn.Dropout(rate)\n",
    "\n",
    "    # x [b, targ_seq_len]\n",
    "    # look_ahead_mask [b, 1, targ_seq_len, targ_seq_len] 这里传入的look_ahead_mask应该是已经结合了look_ahead_mask和padding mask的mask\n",
    "    # enc_output [b, inp_seq_len, d_model]\n",
    "    # padding_mask [b, 1, 1, inp_seq_len]\n",
    "    def forward(self, x, enc_output, look_ahead_mask, padding_mask):\n",
    "        targ_seq_len = x.shape[-1]\n",
    "\n",
    "        attention_weights = {}\n",
    "\n",
    "        # adding embedding and position encoding\n",
    "        x = self.embedding(x)  # [b, targ_seq_len]=>[b, targ_seq_len, d_model]\n",
    "        # 缩放 embedding 原始论文的3.4节有提到： In the embedding layers, we multiply those weights by \\sqrt{d_model}.\n",
    "        x *= torch.sqrt(torch.tensor(self.d_model, dtype=torch.float32))\n",
    "        # x += self.pos_encoding[:, :targ_seq_len, :]  # [b, targ_seq_len, d_model]\n",
    "        pos_encoding = self.pos_encoding[:, :targ_seq_len, :]  # [b, targ_seq_len, d_model]\n",
    "        pos_encoding = pos_encoding.cuda() # ###############\n",
    "        x += pos_encoding  # [b, inp_seq_len, d_model]\n",
    "\n",
    "        x = self.dropout(x)\n",
    "\n",
    "        for i in range(self.num_layers):\n",
    "            x, attn_block1, attn_block2 = self.dec_layers[i](x, enc_output, look_ahead_mask, padding_mask)\n",
    "            # => [b, targ_seq_len, d_model], [b, num_heads, targ_seq_len, targ_seq_len], [b, num_heads, targ_seq_len, inp_seq_len]\n",
    "\n",
    "            attention_weights[f'decoder_layer{i + 1}_block1'] = attn_block1\n",
    "            attention_weights[f'decoder_layer{i + 1}_block2'] = attn_block2\n",
    "\n",
    "        return x, attention_weights\n",
    "        # => [b, targ_seq_len, d_model],\n",
    "        # {'..block1': [b, num_heads, targ_seq_len, targ_seq_len],\n",
    "        #  '..block2': [b, num_heads, targ_seq_len, inp_seq_len], ...}\n",
    "\n",
    "sample_decoder = Decoder(num_layers=2,\n",
    "                         d_model=512,\n",
    "                         num_heads=8,\n",
    "                         dff=2048,\n",
    "                         target_vocab_size=8000,\n",
    "                         maximun_position_encoding=5000)\n",
    "sample_decoder = sample_decoder.to(device)\n",
    "\n",
    "y = torch.tensor(np.random.randint(low=0, high=200, size=(64, 36))) # [b, seq_len]\n",
    "# print(y.shape) # [64, 36]\n",
    "output, attn = sample_decoder(y.cuda(),\n",
    "                              enc_output=sample_encoder_output, # [64, 42, 512]\n",
    "                              look_ahead_mask=None,\n",
    "                              padding_mask=None)\n",
    "print(output.shape) # [64, 36, 512]\n",
    "print(attn['decoder_layer2_block1'].shape) # [64, 8, 36, 36]\n",
    "print(attn['decoder_layer2_block2'].shape) # [64, 8, 36, 42]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 11.搭建transformer\n",
    "transformer 含有3各部分\n",
    "- encoder\n",
    "- decoder\n",
    "- final linear layer"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 36, 8000])\n",
      "torch.Size([64, 8, 36, 36])\n",
      "torch.Size([64, 8, 36, 42])\n"
     ]
    }
   ],
   "source": [
    "class Transformer(torch.nn.Module):\n",
    "    def __init__(self,\n",
    "                 num_layers,  # N个encoder layer\n",
    "                 d_model,\n",
    "                 num_heads,\n",
    "                 dff,  # 点式前馈网络内层fn的维度\n",
    "                 input_vocab_size,  # input此表大小（源语言（法语））\n",
    "                 target_vocab_size,  # target词表大小（目标语言（英语））\n",
    "                 pe_input,  # input max_pos_encoding\n",
    "                 pe_target,  # input max_pos_encoding\n",
    "                 rate=0.1):\n",
    "        super(Transformer, self).__init__()\n",
    "\n",
    "        self.encoder = Encoder(num_layers,\n",
    "                               d_model,\n",
    "                               num_heads,\n",
    "                               dff,\n",
    "                               input_vocab_size,\n",
    "                               pe_input,\n",
    "                               rate)\n",
    "        self.decoder = Decoder(num_layers,\n",
    "                               d_model,\n",
    "                               num_heads,\n",
    "                               dff,\n",
    "                               target_vocab_size,\n",
    "                               pe_target,\n",
    "                               rate)\n",
    "        self.final_layer = torch.nn.Linear(d_model, target_vocab_size)\n",
    "\n",
    "    # inp [b, inp_seq_len]\n",
    "    # targ [b, targ_seq_len]\n",
    "    # enc_padding_mask [b, 1, 1, inp_seq_len]\n",
    "    # look_ahead_mask [b, 1, targ_seq_len, targ_seq_len]\n",
    "    # dec_padding_mask [b, 1, 1, inp_seq_len] # 注意这里的维度是inp_seq_len\n",
    "    def forward(self, inp, targ, enc_padding_mask, look_ahead_mask, dec_padding_mask):\n",
    "        enc_output = self.encoder(inp, enc_padding_mask)  # =>[b, inp_seq_len, d_model]\n",
    "\n",
    "        dec_output, attention_weights = self.decoder(targ, enc_output, look_ahead_mask, dec_padding_mask)\n",
    "        # => [b, targ_seq_len, d_model],\n",
    "        # {'..block1': [b, num_heads, targ_seq_len, targ_seq_len],\n",
    "        #  '..block2': [b, num_heads, targ_seq_len, inp_seq_len], ...}\n",
    "        final_output = self.final_layer(dec_output)  # =>[b, targ_seq_len, target_vocab_size]\n",
    "\n",
    "        return final_output, attention_weights\n",
    "        # [b, targ_seq_len, target_vocab_size]\n",
    "        # {'..block1': [b, num_heads, targ_seq_len, targ_seq_len],\n",
    "        #  '..block2': [b, num_heads, targ_seq_len, inp_seq_len], ...}\n",
    "\n",
    "\n",
    "sample_transformer = Transformer(num_layers=2,\n",
    "                                 d_model=512,\n",
    "                                 num_heads=8,\n",
    "                                 dff=2048,\n",
    "                                 input_vocab_size=8500,\n",
    "                                 target_vocab_size=8000,\n",
    "                                 pe_input=10000,\n",
    "                                 pe_target=6000)\n",
    "sample_transformer = sample_transformer.to(device)\n",
    "\n",
    "temp_inp = torch.tensor(np.random.randint(low=0, high=200, size=(64, 42))) # [b, inp_seq_len]\n",
    "temp_targ = torch.tensor(np.random.randint(low=0, high=200, size=(64, 36))) # [b, targ_seq_len]\n",
    "\n",
    "fn_out, attn = sample_transformer(temp_inp.cuda(), temp_targ.cuda(), None, None, None)\n",
    "print(fn_out.shape) # [64, 36, 8000]\n",
    "print(attn['decoder_layer2_block1'].shape) # [64, 8, 36, 36]\n",
    "print(attn['decoder_layer2_block2'].shape) # [64, 8, 36, 42]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 12.设置超参"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3901 2591\n"
     ]
    }
   ],
   "source": [
    "#Transformer 的基础模型使用的数值为：num_layers=6，d_model = 512，dff = 2048\n",
    "#为了让本示例小且相对较快，已经减小了num_layers、 d_model 和 dff 的值。\n",
    "num_layers = 4\n",
    "d_model = 128\n",
    "dff = 512\n",
    "num_heads = 8\n",
    "\n",
    "input_vocab_size = len(SRC_TEXT.vocab) # 3901\n",
    "target_vocab_size = len(TARG_TEXT.vocab) # 2591\n",
    "dropout_rate = 0.1\n",
    "\n",
    "print(input_vocab_size, target_vocab_size)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 13.优化器\n",
    "根据论文中的公式，将 Adam 优化器与自定义的学习速率调度程序（scheduler）配合使用。\n",
    "\n",
    "$$lrate = d_{model}^{-0.5} * min(\\text{step_num}^{-0.5},\\; \\text{step_num}*\\text{warmup_steps}^{-1.5})$$\n",
    "\n",
    "关于pytorch optimizer，参考：\n",
    "\n",
    "https://www.cnblogs.com/wanghui-garcia/p/10895397.html\n",
    "\n",
    "https://www.jianshu.com/p/5d85a59f1bac"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "outputs": [],
   "source": [
    "class CustomSchedule(torch.optim.lr_scheduler._LRScheduler):\n",
    "    def __init__(self, optimizer, d_model, warm_steps=4):\n",
    "        self.optimizer = optimizer\n",
    "        self.d_model = d_model\n",
    "        self.warmup_steps = warm_steps\n",
    "\n",
    "        super(CustomSchedule, self).__init__(optimizer)\n",
    "\n",
    "    def get_lr(self):\n",
    "        \"\"\"\n",
    "        # rsqrt 函数用于计算 x 元素的平方根的倒数.  即= 1 / sqrt{x}\n",
    "        arg1 = torch.rsqrt(torch.tensor(self._step_count, dtype=torch.float32))\n",
    "        arg2 = torch.tensor(self._step_count * (self.warmup_steps ** -1.5), dtype=torch.float32)\n",
    "        dynamic_lr = torch.rsqrt(self.d_model) * torch.minimum(arg1, arg2)\n",
    "        \"\"\"\n",
    "        # print('*'*27, self._step_count)\n",
    "        arg1 = self._step_count ** (-0.5)\n",
    "        arg2 = self._step_count * (self.warmup_steps ** -1.5)\n",
    "        dynamic_lr = (self.d_model ** (-0.5)) * min(arg1, arg2)\n",
    "        # print('dynamic_lr:', dynamic_lr)\n",
    "        return [dynamic_lr for group in self.optimizer.param_groups]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\n",
      "  \"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\", UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": "<Figure size 432x288 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEGCAYAAACtqQjWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA5tUlEQVR4nO3deXwV5fX48c/JQiAQAoRAIAmEJSxhE4gsCrQKCC41bd1QW+Fb/dpa/dqqtdVfFy3VVtRqtXUp1gWtVdDWirtBVFDZAqJCwhL2sCQhLCEggYTz+2Mm8RJuwk1yb25y73m/Xnkx95mZZ87chHvuzDNzRlQVY4wxxh8igh2AMcaY0GFJxRhjjN9YUjHGGOM3llSMMcb4jSUVY4wxfhMV7ACCqXPnzpqWlhbsMIwxpkVZuXLlXlVN9DYvrJNKWloaOTk5wQ7DGGNaFBHZVts8O/1ljDHGbyypGGOM8RtLKsYYY/wmrMdUjAlHx48fp6CggKNHjwY7FNPMtW7dmpSUFKKjo31ex5KKMWGmoKCAuLg40tLSEJFgh2OaKVWlpKSEgoICevXq5fN6AT39JSJTRWS9iOSLyB1e5seIyFx3/jIRSfOYd6fbvl5Epni0PyMiRSKyppZt3iYiKiKdA7JTxrRwR48eJSEhwRKKqZOIkJCQUO8j2oAlFRGJBB4DzgcygCtFJKPGYtcC+1W1L/AwMMtdNwOYBgwCpgKPu/0BPOe2edtmKnAesN2vO2NMiLGEYnzRkL+TQB6pjALyVXWzqh4DXgayaiyTBcxxp18FJoqzF1nAy6parqpbgHy3P1R1EbCvlm0+DPwSsHr+frZy236Wbi4JdhjGmGYukEklGdjh8brAbfO6jKpWAAeBBB/XPYmIZAE7VfWL0yx3vYjkiEhOcXGxL/thgEue+Ixps5ey88DXwQ7FmGajtLSUlJQUbrrppuq2lStXMmTIEPr27cvNN99M1TOr9u3bx+TJk0lPT2fy5Mns378fcMYubr75Zvr27cvQoUNZtWqVz9v/y1/+wpEjR/y7U40UEpcUi0gs8P+A351uWVWdraqZqpqZmOi1yoCpIb+orHr61699hT3YzTQHlZWVwQ6B3/72t0yYMOGkthtuuIGnnnqKjRs3snHjRt59910A7rvvPiZOnMjGjRuZOHEi9913HwDvvPNO9bKzZ8/mhhtu8Hn74ZZUdgKpHq9T3Davy4hIFBAPlPi4rqc+QC/gCxHZ6i6/SkSSGhG/cWXnFgJw/YTefLS+mPlf7ApyRKYle+CBB3j00UcBuOWWWzj33HMBWLhwIVdffTXgfDBnZmYyaNAg7rrrrup109LS+NWvfsWIESN45ZVXSEtL48477+SMM84gMzOTVatWMWXKFPr06cOTTz4JwEcffcRFF11U3cdNN93Ec889V93fL3/5S4YMGcKoUaPIz8/3eT9WrlxJYWEh5513XnXb7t27KS0tZcyYMYgI11xzDf/9738BeP3115k+fToA06dPP6n9mmuuQUQYM2YMBw4cYPfu3Sdt6/Dhw1x44YUMGzaMwYMHM3fuXB599FF27drFOeecwznnnAPA+++/z9ixYxkxYgSXXXYZZWVlde7nK6+8wuDBgxk2bNgpybGhAnlJ8QogXUR64SSEacBVNZaZD0wHlgCXAgtVVUVkPvAvEXkI6A6kA8tr25CqfgV0qXrtJpZMVd3rv90JX9m5exic3J5fTR3A8i37+P0buYxPT6RT21bBDs000u/fWEvurlK/9pnRvT13fWdQrfPHjx/Pn//8Z26++WZycnIoLy/n+PHjLF68uPqD7d5776VTp05UVlYyceJEvvzyS4YOHQpAQkJC9SmiO+64gx49erB69WpuueUWZsyYwaeffsrRo0cZPHgwP/nJT04bb3x8PF999RXPP/88P//5z3nzzTd58cUXeeCBB05Ztm/fvrz66qucOHGC2267jX/+858sWLCgev7OnTtJSUmpfp2SksLOnc734cLCQrp16wZAUlIShYWF1eukpqaesk7VsgDvvvsu3bt356233gLg4MGDxMfH89BDD/Hhhx/SuXNn9u7dyz333MOCBQto27Yts2bN4qGHHuJ3v/tdrfs5c+ZM3nvvPZKTkzlw4MBp3ytfBOxIxR0juQl4D8gD5qnqWhGZKSIXu4s9DSSISD5wK3CHu+5aYB6QC7wL3KiqlQAi8hJOEuovIgUicm2g9sFA8aFyPt9xgMkDk4iMEGZdMpTSr4/zhzdzgx2aaaFGjhzJypUrKS0tJSYmhrFjx5KTk8PixYsZP348APPmzWPEiBEMHz6ctWvXkpv7zd/bFVdccVJ/F1/sfJwMGTKE0aNHExcXR2JiIjExMT59UF555ZXV/y5ZsgSAq6++mtWrV5/y8+qrrwLw+OOPc8EFF5yUQOpDROp1ZdWQIUPIzs7mV7/6FYsXLyY+Pv6UZZYuXUpubi5nn302Z5xxBnPmzGHbtm/qPnrbz7PPPpsZM2bw1FNP+e10YkBvflTVt4G3a7T9zmP6KHBZLeveC9zrpf1KH7abVt9YjXcL1xWiCpMzugLQPymOn367D48uzOfCId2Y5LablqmuI4pAiY6OplevXjz33HOcddZZDB06lA8//JD8/HwGDhzIli1bePDBB1mxYgUdO3ZkxowZJ90r0bZt25P6i4mJASAiIqJ6uup1RUUFUVFRnDhxorq95n0Xnh/uVdOnO1JZsmQJixcv5vHHH6esrIxjx47Rrl07fvazn1FQUFC9fEFBAcnJzjVGXbt2Zffu3XTr1o3du3fTpYtzciU5OZkdO3Z4XadKv379WLVqFW+//Ta/+c1vmDhxYvURSBVVZfLkybz00kunxF3bfj755JMsW7aMt956qzrZJyQkeF3fVyExUG8CJzu3kOQObRjYLa667cZz+zKwW3vu+M+X7C0rD2J0pqUaP348Dz74IBMmTGD8+PE8+eSTDB8+HBGhtLSUtm3bEh8fT2FhIe+8806jttWzZ09yc3MpLy/nwIEDfPDBByfNnzt3bvW/Y8eOBU5/pPLiiy+yfft2tm7dyoMPPsg111zDfffdR7du3Wjfvj1Lly5FVXn++efJynLupLj44ouZM8e5g2LOnDkntT///POoKkuXLiU+Pv6kU18Au3btIjY2lh/84Afcfvvt1af/4uLiOHToEABjxozh008/rR4vOXz4MBs2bKhzPzdt2sTo0aOZOXMmiYmJJyW3hrIyLaZWR45VsHjjXq4c1eOkbzkxUZH85Yoz+M5fP+HO/3zF7B+OtJvpTL2MHz+ee++9l7Fjx9K2bVtat25dfepr2LBhDB8+nAEDBpCamsrZZ5/dqG2lpqZy+eWXM3jwYHr16sXw4cNPmr9//36GDh1KTExMrd/y6+Pxxx9nxowZfP3115x//vmcf/75gDP+c/nll/P000/Ts2dP5s2bB8AFF1zA22+/Td++fYmNjeXZZ589pc+vvvqK22+/nYiICKKjo3niiScAuP7665k6dSrdu3fnww8/5LnnnuPKK6+kvNz5snfPPffQr1+/Wvfz9ttvZ+PGjagqEydOZNiwYY3efwnny0MzMzPVHtJVu/fW7uHHL6zkxetGc3bfU6ve/GPxZu55K4/7LxnK5WemeunBNEd5eXkMHDgw2GE0C1UP6uvcObSrOjVmP739vYjISlXN9La8nf4ytcrOLaR96yhG9erkdf6Pzu7F2N4J/P6NtWwvaV7XyhtjgsOSivGq8oSycF0R5wzoQnSk9z+TiAjhwcuHEREh/Gzu5xyvPOF1OWOaq61bt4b8UQo07X5aUjFerdq+n32Hj1Vf9VWb5A5tuO/7Q/l8+wHuf3ddE0VnGiucT3sb3zXk78SSivEqO7eQ6EjhW/1OX8rmwqHduGZsT55avKX67nvTfLVu3ZqSkhJLLKZOVc9Tad26db3Ws6u/zClUlezcQsb0TiCutW9PfPt/Fwxk1fb9/OKVL3jr5nGkdIwNcJSmoVJSUigoKMAKqprTqXryY31YUjGn2FRcxpa9h/nR2Wk+r9M6OpLHrhrBRY9+wk3/+px5Px5Lqyg7EG6Oqm4+NCYQ7H+9OcX77ims+t4t3zOhLfdfOpTVOw4w8821gQjNGNPMWVIxp1iQW8iQ5Hi6xbep97rnD+nGT77Vh38u3c6Ly7adfgVjTEixpGJOUl1AshE1vW6f0p9v90/krtfXsnxLbQ/pNMaEIksq5iQf5DkFJCcNbHhSiYwQHpk2nB6dYrnhnyvtaZHGhBFLKuYk3gpINkR8m2hmX5PJsYoTXP98DofLK/wUoTGmObOkYqodOVbBJ/l7mZzR1S8FIvt2acejVw4nb3cp//fS51TYHffGhDxLKqba4o17Ka84wXl+fEbKOQO68IfvDmbhuiLumr/WbrgzJsTZfSqmWlUByTNrKSDZUFeP7knB/q954qNNpHSM5YZv9/Fr/8aY5sOSigF8KyDZGLef15+d+79m1rvr6N6hNVlnJJ9+JWNMi2NJxQCwcptvBSQbKiJCeOCyoRSWHuUXr3xB+zbRnNO/S0C2ZYwJnoCOqYjIVBFZLyL5InKHl/kxIjLXnb9MRNI85t3ptq8XkSke7c+ISJGIrKnR1wMisk5EvhSR10SkQyD3LdRk5+7xuYBkQ8VERfLU9Ez6J8XxkxdWsnRzScC2ZYwJjoAlFRGJBB4DzgcygCtFJKPGYtcC+1W1L/AwMMtdNwOYBgwCpgKPu/0BPOe21ZQNDFbVocAG4E6/7lAIqyogObZPZ58LSDZU+9bRzPmfUaR2iuW6OTl8seNAQLdnjGlagTxSGQXkq+pmVT0GvAxk1VgmC5jjTr8KTBTnWtYs4GVVLVfVLUC+2x+qugg45TZtVX1fVatuhlgK1K+0ZhjbVFzG1pIjATv1VVNCuxj+ee1oOraNZvqzy1m/51CTbNcYE3iBTCrJwA6P1wVum9dl3IRwEEjwcd26/Ah4x9sMEbleRHJEJMdKfzuqC0gObLoxjqT41rx47RhioiK4+h/L2FhoicWYUBBy96mIyK+BCuBFb/NVdbaqZqpqZmJi4MYPWpLsRhSQbIweCbG8eN0YRGDa7KV2xGJMCAhkUtkJpHq8TnHbvC4jIlFAPFDi47qnEJEZwEXA1Wp32fmk6NBRVjeygGRj9O3SjpevH0NUpDBt9hJyd5UGJQ5jjH8EMqmsANJFpJeItMIZeJ9fY5n5wHR3+lJgoZsM5gPT3KvDegHpwPK6NiYiU4FfAher6hE/7kdIW5hXhCpBSyoAfRLbMff6sbSOjuSqfyxlzc6DQYvFGNM4AUsq7hjJTcB7QB4wT1XXishMEbnYXexpIEFE8oFbgTvcddcC84Bc4F3gRlWtBBCRl4AlQH8RKRCRa92+/gbEAdkislpEngzUvoWS7NxCUjq2YUBS4wpINlZa57bMvX4sbVtFcdVTS1m5zUrmG9MSSTifJcrMzNScnJxghxE0R45VMHxmNleO6sHdFw8KdjgAFOw/wg+fXs7ug1/z+NUjOHdA8I6gjDHeichKVc30Ni/kBuqN7xZt8H8BycZK6RjLKz8ZS3qXOP73+ZX8e2VBsEMyxtSDJZUwFqgCko3VuV0ML10/hjG9O3HbK18we9GmYIdkjPGRJZUw5RSQLOTcABWQbKx2MVE8M+NMLhzajT++vY4/vJlL5YnwPVVrTEthBSXD1Mpt+9l/5DiTM5KCHUqtYqIieXTacBLbxfD0J1vYVnKYR6YNp22M/dka01w1v6+opklUFZCc0K9zsEOpU2SEcPfFg5iZNYiF64q49Mkl7LJn3hvTbFlSCUNNWUDSX64Zm8YzM85kx74jfPexT/my4ECwQzLGeGFJJQzlFzVtAUl/+Xb/Lvz7hrOIjozg8r8v4b+fn7bIgjGmiVlSCUPZeU4ByckDW1ZSAeifFMd/bzybockd+Pnc1dw9fy3HK08EOyxjjMuSShjKzi1kaEo8SfGtgx1KgyTGxfDi/47mf85O47nPtnLVU0spKj0a7LCMMVhSCTtVBSQntcCjFE/RkRHc9Z1BPDLtDNbsLOWiv35CzlYr7WJMsFlSCTMfNIMCkv6UdUYyr914Fm1aRTJt9lKe+GgTJ+x+FmOCxpJKmGkuBST9aUBSe+bfNI7zBnVl1rvrmP7scooO2ekwY4LBkkoYOXKsgk/y9zI5oyvOU5tDR3ybaB67agR//N4Qlm/ZxwWPLGbRBnuypzFNzZJKGFm0YS/HKk6EzKmvmkSEq0b3YP5N4+jUthXXPLOcP72dR3lFZbBDMyZsWFIJI9UFJNOaVwFJf+ufFMfrN47jqtE9+PuizWT97VN7oqQxTcSSSpioqDzRrAtI+lubVpH88XtDeHp6JiWHj5H12Cf8beFGKuyeFmMCKvQ/XQzQMgpIBsLEgV15/+cTmDq4Gw++v4FLnviM/KJDwQ7LmJBlSSVMLMgrpFVkBN/qnxjsUJpcx7at+OuVw3nsqhFs33eECx79hCc+2mR34hsTAJZUwsA3BSQTaBfGZeMvHNqN92/5Fuf0T2TWu+v4zl8/YfWOA8EOy5iQEtCkIiJTRWS9iOSLyB1e5seIyFx3/jIRSfOYd6fbvl5Epni0PyMiRSKypkZfnUQkW0Q2uv92DOS+tSRVBSQnhehVX/WRGBfD33+Yyd9/OJIDR47zvcc/5e75azl09HiwQzMmJAQsqYhIJPAYcD6QAVwpIhk1FrsW2K+qfYGHgVnuuhnANGAQMBV43O0P4Dm3raY7gA9UNR34wH1tgPdzW24ByUCZMiiJ7FsncM2YnsxZspXJDy3ivbV7ULW78Y1pjEAeqYwC8lV1s6oeA14GsmoskwXMcadfBSaKc1deFvCyqpar6hYg3+0PVV0EeCvy5NnXHOC7ftyXFq2lF5AMlLjW0fw+azD/ueEsOsRG8+MXVjLj2RVsKi4LdmjGtFiBTCrJwA6P1wVum9dlVLUCOAgk+LhuTV1Vdbc7vQewr+V8U0DSjlJqN7xHR974v3H89qIMVm3bz5SHF/HHt/PslJgxDRCSA/XqnMPweh5DRK4XkRwRySkuDv0yHh/kFQEweZAllbpER0Zw7bheLPzFt/n+iGSeWryZc//8Mf9eWWAFKo2ph0AmlZ1AqsfrFLfN6zIiEgXEAyU+rltToYh0c/vqBhR5W0hVZ6tqpqpmJiaG/uW12bmFpHZqQ/+uoVNAMpAS42K4/9Jh/PenZ5PcoQ23vfIF33/iM1ZYWX1jfBLIpLICSBeRXiLSCmfgfX6NZeYD093pS4GF7lHGfGCae3VYLyAdWH6a7Xn2NR143Q/70KIdLncKSE4aGHoFJANtWGoH/nPDWTx42TB2H/yay55cwnVzcuzGSWNOI2BJxR0juQl4D8gD5qnqWhGZKSIXu4s9DSSISD5wK+4VW6q6FpgH5ALvAjeqaiWAiLwELAH6i0iBiFzr9nUfMFlENgKT3NdhbfHG4pAuIBloERHCpSNT+OgX53D7lP4s3VzCeQ8v4s7/fEmhPWnSGK8knC+hzMzM1JycnGCHETC3zlvNB3lFrPzNJKLCoN5XoJWUlfPXhfm8uGwbkRHCdeN6878TehPfJjrYoRnTpERkpapmeptnnzQhqqLyBB+uK+LcAV0sofhJQrsY7r54EB/c+m0mZyTxtw/zGTdrIY8s2EipXSlmDGBJJWR9U0DSTn35W4+EWP565XDeunkcY3on8PCCDYyf9SF/W7iRsvKKYIdnTFBZUglR2blOAckJ/UL/CrdgGdQ9nqeuyeSNm8aR2bMjD76/gXGzFvL4R/kctuRiwpQllRCkqmTnWQHJpjIkJZ6nZ5zJ6zeezfDUDtz/7nrOnrWQh7M3sP/wsWCHZ0yTsqQSgjYWlbGt5Iid+mpiw1I78Oz/jOK1n57FmWmdeOSDjZx130J+/8Zadh34OtjhGdMkTptURKSfiHxQVRVYRIaKyG8CH5ppqOyqApKWVIJieI+OPHVNJtm3TOCCId14Yck2Jtz/Ib945Qu7z8WEPF+OVJ4C7gSOA6jqlzg3MppmKju3kGEp8XRtbwUkgym9axx/vnwYH93+bX4wpidvfrmLSQ8t4ro5K/gsf69VRDYhyZekEquqNe9mt1HIZqqo1CkgOckKSDYbKR1jufviQXx2x0RunpjOqu0HuOofyzj/kcXMXbGdo8crgx2iMX7jS1LZKyJ9cAs0isilwO66VzHBssAKSDZbndq24tbJ/fjsjnO5/5KhAPzq318x9k8f8OB76+0ufRMSfLk06EZgNjBARHYCW4CrAxqVabDs3D1WQLKZax0dyeVnpnJZZgpLNpfw7KdbeeyjfJ78eBMXDOnGD8b05My0jlavzbRIviQVVdVJItIWiFDVQ26RR9PMHC6v4NNNJfxgdE/7QGoBRISz+nTmrD6d2VZymDmfbeOVnB3M/2IX6V3acdXoHnx/eArxsVYGxrQcvpz++jeAqh5W1apLV14NXEimoayAZMvVM6Etv/tOBst+PZH7LxlKbEwUv38jl1F/XMBt875g1fb9NrBvWoRaj1REZADOM+LjReT7HrPaA3ZZUTP0fm4h8W2iOTOtY7BDMQ0U2yqKy89M5fIzU1mz8yD/Wr6d1z/fyb9XFTAgKY6rRvfg4mHd6RDbKtihGuNVrVWKRSQL5znvF3Pyc1AO4Tw//rOARxdgoVSluKLyBJn3LuCc/l14+Iozgh2O8aOy8gpeX72Tfy3bztpdpbSKjGBSRhcuHZnChPREKxhqmlxdVYprPVJR1deB10VkrKouCVh0xi9ytu3ngBWQDEntYqK4enRPrh7dk7W7DvLqygJeX72Lt7/aQ+d2MXxveHcuHZlK/yS7OMMEny8D9Z+LyI04p8KqT3up6o8CFpWptwVWQDIsDOoez6Du8dx5/kA+Wl/EqysLePbTrTy1eAtDkuO5ZEQyFw7tTmJcTLBDNWHKl6TyArAOmALMxLmcOC+QQZn6qSogeVZfKyAZLlpFRXDeoCTOG5RESVk5r6/exasrC7j7jVxmvpnLWX06c/Gw7kwZnGQPETNN6rRPfhSRz1V1uIh8qapDRSQaWKyqY5omxMAJlTGVDYWHOO/hRdzz3cH8YEzPYIdjgmhD4SHmr97F/C92sX3fEVpFRvCt/ol8Z1h3Jg3sQmwr+9JhGq9BYyoeqh5pd0BEBgN7gC7+Cs40nhWQNFX6dY3jF1P6c9t5/fiy4CDzv9jFm1/uIju3kNhWkUwa2JWLhnZjQr9EWkdHBjtcE4J8SSqzRaQj8Bucq8DaAb8NaFSmXt63ApKmBhFhWGoHhqV24P9dMJDlW/bxxpe7eOer3cz/YhexrSI5p38XpgxO4pz+icS1tlNkxj9Om1RU9R/u5CKgN4CI9PClcxGZCjwCRAL/UNX7asyPAZ4HRgIlwBWqutWddydwLVAJ3Kyq79XVp4hMBB7AuaGzDJihqvm+xNmSFZUe5YsdB/jFef2CHYpppiIjhLF9EhjbJ4HfXzyIZZv38c6a3by3tpC3vtpNq8gIxqV3ZuqgJCZldKVTW7sHxjRcnUlFRMYCycAiVS0SkaHAHcB4IPU060YCjwGTgQJghYjMV9Vcj8WuBfaral8RmQbMAq4QkQyc8vqDgO7AAhGp+tSsrc8ngCxVzRORn+IcWc3w9Y1oqaoLSGYkBTkS0xJEuwlkXHpnZmYN5vPt+3l3zR7eWbOHheuKiHxNGN2rE1MGJXHugC6kdooNdsimhanrjvoHgIuA1cCvROQ94DrgT4AvlxOPAvJVdbPb38tAFuCZVLKAu93pV4G/iVO0KgvnBstyYIuI5Lv9UUefinO3P0A8sMuHGFu8qgKS/bq2C3YopoWJjBAy0zqRmdaJX184kLW7St0Es5u75q/lrvlr6d81jnMHdmHigC4M79GRyAirKWfqVteRyoXAcFU96o6p7AAGV52e8kGyu06VAmB0bcuoaoWIHAQS3PalNdZNdqdr6/M64G0R+RooBbxenSYi1wPXA/To4dNZvGbLCkgafxERBifHMzg5nl9M6c/m4jIWriti4boinlq0mSc+2kTH2Gi+3b8L5w7owoR+iXapsvGqrqRyVFWPAqjqfhHZWI+EEgy3ABeo6jIRuR14CCfRnERVZ+OU8iczM7NFV+hbtMEKSJrA6J3Yjt6J7bhufG9Kjx5n8Ya9fJBXyIfri3jt851ERghnpnXknP5OghmQFGdfbAxQd1LpLSKeNb96eb5W1YtP0/dOTh53SXHbvC1TICJROKetSk6z7intIpIIDFPVZW77XODd08TX4mXnFdIh1gpImsBq3zqaC4d248Kh3ag8oazesZ8P8pyjmD+9s44/vbOOxLgYxqd3ZkJ6IuPSO9O5nd3RH67qSipZNV7/uZ59rwDS3Wev7MQZeL+qxjLzgenAEuBSYKGqqpu8/iUiD+EM1KcDywGppc/9ONWU+6nqBpyB/JC+67+i8gQL1xVxbv8uVlDQNJnICGFkz06M7NmJX04dwJ6DR1m8sZhFG/fy4boi/rPK+e43qHt7xqcnMiG9MyPTOhITZffEhIu6Ckp+3JiO3TGSm4D3cC7/fUZV14rITCBHVecDTwMvuAPx+3CSBO5y83AG4CuAG1W1EsBbn277/wL/FpETOEkmpGuTVRWQnGSnvkwQJcW35rLMVC7LTOXECWXtrlIWbSxm0YZi/rF4M09+vIk20ZGM7t2Js/okMLZ3ZzK6t7cB/xB22jItoawll2n5w5u5vLBkG6t+N9nqfZlmqay8gmWbS1i0oZjF+XvZXHwYgPatoxjVy7lvZmzvBAYkxRFhSaZFaWyZFtPMqCrZuVZA0jRv7WKimDiwKxMHOkfThaVHWbq5hCWbSliyuYQFeU55oY6x0YyuSjJ9Ekjv0s4G/Vsw+0RqgTYWlbF93xF+/K3ewQ7FGJ91bd+arDOSyTrDuTtg14GvqxPMkk0lvLt2DwCd27XiTPf+mTPTOpLRrb2NG7Ygp00qIvIGzo2Fng4COcDfqy47Nk2nqoDkpIE2nmJaru4d2nDJyBQuGZkCwI59R1iyqYSlm0tYsW0f76xxkkxsq0iG9+hAZs9OnJnWieE9OtDWjtCbLV9+M5uBROAl9/UVOI8U7gc8BfwwMKGZ2lgBSROKUjvFktoplsvPdO4a2HPwKDnb9pGzdT8rtu7jrws3ckKdK9AyurUnM62jc0TTsyNd7P9Cs+FLUjlLVc/0eP2GiKxQ1TNFZG2gAjPeFVoBSRMmkuJbc9HQ7lw0tDsAh44e5/PtB8jZuo8VW/fz0vLtPPvpVgCSO7ThjB4dGJ7ageE9OjCoe7yV9g8SX5JKOxHpoarbobpCcVWhqWMBi8x4VTW4aQUkTbiJax3NhH6J1Y/MPl55grW7SsnZuo/Pdxxg9fYDvPXlbgCiIoSB3dozvEcHzkh1fnp1bmsXADQBX5LKbcAnIrIJ5+bDXsBPRaQtMCeQwZlTLcgtpEenWCsgacJedGREdcKoUnToKKu3H2D1jgN8vv0A/15ZwPNLtgHQITaaYSlukunRgSHJ8XbnfwD48jyVt0UkHRjgNq33GJz/S6ACM6eqKiD5wzFWQNIYb7rEtea8QUmcN8g5kq88oeQXlfH59v3ViebRjRupuj2vW3xrBifHM8T9GZwcT2KcJZrG8PUSipFAmrv8MBFBVZ8PWFTGq6oCknbVlzG+iYwQ+ifF0T8pjmmjnKrkZeUVrNl5kDU7D/KV+7Mgr7A60XRtH1OdYKqSjV0I4DtfLil+AeiD81yVSrdZcZ7YaJpQdq4VkDSmsdrFRDGmdwJjeidUt5WVV7DWTTBVyeaDdUXViaZLnJNoMrq3Z2A356dnp1irBOCFL0cqmUCGhnM9l2agovIEC9dbAUljAqFdTBSjeycwukaiyd1Vylc7D1YnnI82FFN5wvkojG0VSf+kuOokk9Etjv5J7cO+yoUve78GSAJ2BzgWU4eqApL27BRjmka7mChG9erEqF6dqtuOHq9kY2EZebtLyd1dSt7uUt78Yhf/Wra9epmeCbEMTKo6onGSTkrHNmEzDupLUukM5IrIcqC8qtGH56kYP8rOLaRVZET15ZTGmKbXOjqSISnxDEmJr25TVXYdPEreLifJ5O0pJW/3Id7L3VN9+iwuJoq+XdvRr0sc6V3b0a9rHP26xtG1fUzIJRtfksrdgQ7C1M2zgKSVpzCmeRERkju0IblDm5MeRXG4vIL1hYfI213K+j2H2FB4iAV5hczN+eaJ6O1bR9GvaxzpXePo5yab9K7tSGzXcpONL5cUN+q5KqbxNhRaAUljWpq2MVGM6NGRET1OvrCmpKycDYVlbCw6xPo9h9hYWMY7a3bz0vLj1ct0iI12j2acRNPXfbxzSziyqTWpiMgnqjpORA5xckFJAVRV2wc8OgNAdq5TWM8uJTam5UtoF8PYdjGM7fPNRQGqSnFZORsLy9hQeMj9KeP11bs4dLSierm2rSLpndiOPolt6eMmmj5d2pKW0LbZlKWp68mP49x/45ouHONNdl4Rw1I7WAFJY0KUiNAlrjVd4lpzdt/O1e2qSmFpOZuLy9hUXMam4sNsKi5jxdb9/Hf1Lo/1IaVjGyfRdHYSTdW/TX0qzacT9CISCXT1XL6qFpgJrKoCkrdP6R/sUIwxTUxESIpvTVJ8a87ySDYAR45VsGXvYTYVH3aTzmE2FZWxbPM+vj5eWb1cXOsoeie2o3fntvTq3Ja0zm3p3bktfbu0C8jRjS83P/4fcBdQCJxwmxUY6vdozCm+KSBpp76MMd+IbRXFoO7xDOoef1L7iRPK7tKjbCoqq042m/eWsXzLPl77fGf1ck9Pz6x+Kqc/+XKk8jOgv6qW1LdzEZkKPAJEAv9Q1ftqzI/BuTN/JFACXKGqW915dwLX4tzFf7OqvldXn+Ic390DXOau84SqPlrfmJubbLeAZHoXKyBpjDm9iIhvrkareQvC18cq2bbvMFuKD59UiNOffEkqO3Ce9Fgv7imzx4DJQAGwQkTmq2qux2LXAvtVta+ITANmAVeISAYwDRgEdAcWiEjVA0Rq63MGkAoMUNUTItKlvjE3N2XlFXyWX8IPx1oBSWNM47VpFcmApPYMSArcdVa+PvnxIxF5i5NvfnzoNOuNAvJVdTOAiLwMZAGeSSWLb+6DeRX4m3vEkQW8rKrlwBYRyXf7o44+bwCuUtUTbnxFPuxbs7ZoQzHHKk/YqS9jTIvhSxGp7UA20AqI8/g5nWSco5wqBW6b12VUtQLniCihjnXr6rMPzlFOjoi845brP4WIXO8uk1NcXOzDbgTPAreAZGZPKyBpjGkZ6jxScU9h9VPVq5sonsaIAY6qaqaIfB94BhhfcyFVnQ3MBsjMzGy2RTKrC0gOsAKSxpiWo85PK1WtBHqKSKsG9L0TZ4yjSorb5nUZEYkC4nEG7Gtbt64+C4D/uNOv0cKvTlux1S0gaTc8GmNaEF/HVD4VkfnA4apGH8ZUVgDpItIL54N/GnBVjWXmA9OBJcClwEJVVXdb/xKRh3AG6tOB5Th389fW53+Bc4AtwLeADT7sW7OVnVtIqygrIGmMaVl8SSqb3J8IfBtLAZwxEhG5CXgP5/LfZ1R1rYjMBHJUdT7wNPCCOxC/DydJ4C43D2cAvgK40T1qwluf7ibvA14UkVuAMuA6X2NtblSV7Lw9nN3HCkgaY1oWCednb2VmZmpOTk6wwzjF+j2HmPKXRfzxe0O4anSPYIdjjDEnEZGVqprpbZ4vd9QnAr/EuWekuviUqp7rtwjNSb4pINnib7UxxoQZXy4rehFYB/QCfg9sxRkvMQGSnVvIsNQOdLECksaYFsaXpJKgqk8Dx1X1Y1X9EWBHKQFSWHqULwoOcp7d8GiMaYF8GQWuenLMbhG5ENgFdKpjedMI2blWQNIY03L5klTuEZF44Dbgr0B74JaARhXGFuQV0jPBCkgaY1omXx4n/KY7eRDnPhATIFUFJK+xApLGmBbqtGMqItJPRD4QkTXu66Ei8pvAhxZ+qgpITrJTX8aYFsqXgfqngDtxx1ZU9UvcmxSNf2VbAUljTAvnS1KJVdXlNdoqAhFMODteeYKF66yApDGmZfPl02uviPTBeYQwInIpsDugUYWhnK37Ofj1cbuU2BjTovly9deNOKXiB4jITpyCjS2hFH6LUlVAcny6FZA0xrRcpz1SUdXNqjoJSMR5VO844HsBjyyMWAFJY0yo8PnkvaoeVtVD7stbAxRPWFpfeIgd+75mckZSsEMxxphGaeiIsN1E4UfZa5276K2ApDGmpWtoUgnfevkBsCCvkDOsgKQxJgTUegJfRA7hPXkI0CZgEYWZqgKSt0/pH+xQjDGm0WpNKqrq81MeTcNZAUljTCixu+yCLDvXCkgaY0KHJZUgKiuvYMmmEiYP7GoFJI0xISGgSUVEporIehHJF5E7vMyPEZG57vxlIpLmMe9Ot329iEypR5+PikhZwHbKj6oKSNqpL2NMqAhYUhGRSOAx4HwgA7hSRDJqLHYtsF9V+wIPA7PcdTNwilYOAqYCj4tI5On6FJFMoMVUY8zOLaRjbDQjrYCkMSZEBPJIZRSQ796Rfwx4GciqsUwWMMedfhWYKM55oCzgZVUtV9UtQL7bX619ugnnAeCXAdwnv6kqIHmOFZA0xoSQQH6aJQM7PF4XuG1el1HVCpwHgSXUsW5dfd4EzFfVOotdisj1IpIjIjnFxcX12iF/WrF1nxWQNMaEnJD4iiwi3YHLcB53XCdVna2qmaqamZgYvOKNVkDSGBOKAplUdgKpHq9T3Davy4hIFBAPlNSxbm3tw4G+QL6IbAViRSTfXzvib6rKgrxCxvXtbAUkjTEhJZBJZQWQLiK9RKQVzsD7/BrLzAemu9OXAgtVVd32ae7VYb2AdGB5bX2q6luqmqSqaaqaBhxxB/+bpW8KSNqpL2NMaAnY12RVrRCRm4D3gEjgGVVdKyIzgRxVnQ88DbzgHlXsw31MsbvcPCAX5ymTN6pqJYC3PgO1D4FSVUBy4gArIGmMCS3iHBiEp8zMTM3JyWny7V78t0+IEOG/N57d5Ns2xpjGEpGVqprpbV5IDNS3JHsOHuXLgoN26ssYE5IsqTSxBXnOqS+7lNgYE4osqTSx7NxC0hJi6WsFJI0xIciSShOqKiA5yQpIGmNClCWVJvTxeisgaYwJbZZUmlB27h4rIGmMCWmWVJpIVQHJcwd0tQKSxpiQZZ9uTWTF1n2UHq2wU1/GmJBmSaWJVBWQnNCvc7BDMcaYgLGk0gRUlexcp4BkbCsrIGmMCV2WVJrAuj2HKNhvBSSNMaHPkkoTyM4tRAQmDrQCksaY0GZJpQksyCvkjNQOdIlrHexQjDEmoCypBJgVkDTGhBNLKgGW7RaQnDzQkooxJvRZUgkwKyBpjAknllQC6NDR4yzZtJfJGVZA0hgTHiypBNCiDXs5XqlMzkgKdijGGNMkLKkEkBWQNMaEG0sqAeJZQDIywk59GWPCQ0CTiohMFZH1IpIvInd4mR8jInPd+ctEJM1j3p1u+3oRmXK6PkXkRbd9jYg8IyLRgdy301mxxQpIGmPCT8CSiohEAo8B5wMZwJUiklFjsWuB/araF3gYmOWumwFMAwYBU4HHRSTyNH2+CAwAhgBtgOsCtW++eD+3kBgrIGmMCTOBPFIZBeSr6mZVPQa8DGTVWCYLmONOvwpMFOcyqSzgZVUtV9UtQL7bX619qurb6gKWAykB3Lc6qSoL8qyApDEm/AQyqSQDOzxeF7htXpdR1QrgIJBQx7qn7dM97fVD4F1vQYnI9SKSIyI5xcXF9dwl31gBSWNMuArFgfrHgUWqutjbTFWdraqZqpqZmJgYkACqCkieawUkjTFhJpDnZnYCqR6vU9w2b8sUiEgUEA+UnGbdWvsUkbuARODHfoi/wbJzrYCkMSY8BfJIZQWQLiK9RKQVzsD7/BrLzAemu9OXAgvdMZH5wDT36rBeQDrOOEmtfYrIdcAU4EpVPRHA/arT7oNf89VOKyBpjAlPATtSUdUKEbkJeA+IBJ5R1bUiMhPIUdX5wNPACyKSD+zDSRK4y80DcoEK4EZVrQTw1qe7ySeBbcAStyTKf1R1ZqD2rzYL8ooAOM+SijEmDIlzYBCeMjMzNScnx699XvPMcnbsO8LC275l9b6MMSFJRFaqaqa3eaE4UB80VQUkJw3sYgnFGBOWLKn40ccbiq2ApDEmrFlS8aPs3EI6tW1lBSSNMWHLkoqfHK88wYfrijh3QBcrIGmMCVuWVPzECkgaY4wlFb+pKiA5Pt0KSBpjwpclFT9QVbJzrYCkMcZYUvGDvN2H2HnACkgaY4wlFT9YkOcUkJw40JKKMSa8WVLxg+zcQoandiAxLibYoRhjTFBZUmmkqgKSk+zUlzHGWFJprAW5hYAVkDTGGLCk0mjv5xbSq3Nb+iS2C3YoxhgTdJZUGuHQ0eMs3VzC5IyuVkDSGGOwpNIo3xSQtFNfxhgDllQapaqA5IgeVkDSGGPAkkqDWQFJY4w5lSWVBlpuBSSNMeYUllQaKNsKSBpjzCkCmlREZKqIrBeRfBG5w8v8GBGZ685fJiJpHvPudNvXi8iU0/UpIr3cPvLdPlsFar+qCkiOT7cCksYY4ylgSUVEIoHHgPOBDOBKEcmosdi1wH5V7Qs8DMxy180ApgGDgKnA4yISeZo+ZwEPu33td/sOiKoCkpOs1pcxxpwkkEcqo4B8Vd2sqseAl4GsGstkAXPc6VeBieLc8JEFvKyq5aq6Bch3+/Pap7vOuW4fuH1+N1A7lp1rBSSNMcabQCaVZGCHx+sCt83rMqpaARwEEupYt7b2BOCA20dt2wJARK4XkRwRySkuLm7AbkFSfAyXjUyxApLGGFND2A3Uq+psVc1U1czExMQG9XHFmT24/9Jhfo7MGGNavkAmlZ1AqsfrFLfN6zIiEgXEAyV1rFtbewnQwe2jtm0ZY4wJsEAmlRVAuntVViucgff5NZaZD0x3py8FFqqquu3T3KvDegHpwPLa+nTX+dDtA7fP1wO4b8YYY7wI2PWwqlohIjcB7wGRwDOqulZEZgI5qjofeBp4QUTygX04SQJ3uXlALlAB3KiqlQDe+nQ3+SvgZRG5B/jc7dsYY0wTEudLfnjKzMzUnJycYIdhjDEtioisVNVMb/PCbqDeGGNM4FhSMcYY4zeWVIwxxviNJRVjjDF+E9YD9SJSDGxr4Oqdgb1+DMdfLK76sbjqx+Kqn+YaFzQutp6q6vXu8bBOKo0hIjm1Xf0QTBZX/Vhc9WNx1U9zjQsCF5ud/jLGGOM3llSMMcb4jSWVhpsd7ABqYXHVj8VVPxZX/TTXuCBAsdmYijHGGL+xIxVjjDF+Y0nFGGOM31hSaQARmSoi60UkX0TuCPC2UkXkQxHJFZG1IvIzt/1uEdkpIqvdnws81rnTjW29iEwJZNwislVEvnJjyHHbOolItohsdP/t6LaLiDzqbv9LERnh0c90d/mNIjK9tu35EE9/j/dktYiUisjPg/V+icgzIlIkIms82vz2/ojISPf9z3fXlUbE9YCIrHO3/ZqIdHDb00Tka4/37snTbb+2fWxgXH773Ynz2IxlbvtccR6h0dC45nrEtFVEVgfh/art8yF4f2Oqaj/1+MEpub8J6A20Ar4AMgK4vW7ACHc6DtgAZAB3A7/wsnyGG1MM0MuNNTJQcQNbgc412u4H7nCn7wBmudMXAO8AAowBlrntnYDN7r8d3emOfvpd7QF6Buv9AiYAI4A1gXh/cJ4zNMZd5x3g/EbEdR4Q5U7P8ogrzXO5Gv143X5t+9jAuPz2uwPmAdPc6SeBGxoaV435fwZ+F4T3q7bPh6D9jdmRSv2NAvJVdbOqHgNeBrICtTFV3a2qq9zpQ0AekFzHKlnAy6parqpbgHw35qaMOwuY407PAb7r0f68OpbiPK2zGzAFyFbVfaq6H8gGpvohjonAJlWtq2pCQN8vVV2E86ygmtts9PvjzmuvqkvV+d//vEdf9Y5LVd9X1Qr35VKcJ6jW6jTbr20f6x1XHer1u3O/YZ8LvOrPuNx+LwdeqquPAL1ftX0+BO1vzJJK/SUDOzxeF1D3h7zfiEgaMBxY5jbd5B7CPuNxuFxbfIGKW4H3RWSliFzvtnVV1d3u9B6ga5Bim8bJ/9Gbw/sF/nt/kt3pQMT4I5xvpVV6icjnIvKxiIz3iLe27de2jw3lj99dAnDAI3H66/0aDxSq6kaPtiZ/v2p8PgTtb8ySSgshIu2AfwM/V9VS4AmgD3AGsBvn8DsYxqnqCOB84EYRmeA50/120+TXrbvnyi8GXnGbmsv7dZJgvT91EZFf4zxx9UW3aTfQQ1WHA7cC/xKR9r7254d9bJa/Ow9XcvKXlyZ/v7x8PjSqv8awpFJ/O4FUj9cpblvAiEg0zh/Mi6r6HwBVLVTVSlU9ATyFc8hfV3wBiVtVd7r/FgGvuXEUuofNVYf8RUGI7XxglaoWuvE1i/fL5a/3Zycnn6JqdIwiMgO4CLja/TDCPb1U4k6vxBmv6Hea7de2j/Xmx99dCc7pnqga7Q3m9vV9YK5HvE36fnn7fKijv8D/jfkyGGQ/Jw2MReEMYvXim0HAQQHcnuCcx/xLjfZuHtO34JxbBhjEyYOXm3EGLv0eN9AWiPOY/gxnLOQBTh4kvN+dvpCTBwmXu+2dgC04A4Qd3elOjYztZeB/msP7RY2BW3++P5w6iHpBI+KaCuQCiTWWSwQi3eneOB8qdW6/tn1sYFx++93hHLl6DtT/tKFxebxnHwfr/aL2z4eg/Y0F5IMw1H9wrqDYgPMN5NcB3tY4nEPXL4HV7s8FwAvAV277/Br/8X7txrYejys1/B23+x/mC/dnbVWfOOeuPwA2Ags8/jgFeMzd/ldApkdfP8IZaM3HIxk0MK62ON9K4z3agvJ+4ZwW2Q0cxzkffa0/3x8gE1jjrvM33CoZDYwrH+e8etXf2ZPuspe4v9/VwCrgO6fbfm372MC4/Pa7c/9ml7v7+goQ09C43PbngJ/UWLYp36/aPh+C9jdmZVqMMcb4jY2pGGOM8RtLKsYYY/zGkooxxhi/saRijDHGbyypGGOM8RtLKsbUk4gkeFSg3SMnV9Cts+qtiGSKyKP13N6P3CqxX4rIGhHJcttniEj3xuyLMf5mlxQb0wgicjdQpqoPerRF6Tf1pRrbfwrwMU4l2oNuOY5EVd0iIh/hVO/N8ce2jPEHO1Ixxg9E5DkReVJElgH3i8goEVniFhX8TET6u8t9W0TedKfvdgskfiQim0XkZi9ddwEOAWUAqlrmJpRLcW5Ke9E9QmrjPvfiY7e453seZTo+EpFH3OXWiMgoL9sxxi8sqRjjPynAWap6K7AOGK9OUcHfAX+sZZ0BOGXHRwF3uXWcPH0BFAJbRORZEfkOgKq+CuTg1Og6A6cA5F+BS1V1JPAMcK9HP7Hucj915xkTEFGnX8QY46NXVLXSnY4H5ohIOk4ZjZrJospbqloOlItIEU6J8upS46paKSJTgTNxng/zsIiMVNW7a/TTHxgMZLsP5ovEKStS5SW3v0Ui0l5EOqjqgYbvqjHeWVIxxn8Oe0z/AfhQVb/nPufio1rWKfeYrsTL/0l1Bj6XA8tFJBt4FudpiJ4EWKuqY2vZTs3BUxtMNQFhp7+MCYx4vikRPqOhnYhId/F4jjjOM0WqnmR5COcRsuAUVEwUkbHuetEiMshjvSvc9nHAQVU92NCYjKmLHakYExj345z++g3wViP6iQYedC8dPgoUAz9x5z0HPCkiXwNjgUuBR0UkHuf/9l9wquUCHBWRz93+ftSIeIypk11SbEyIs0uPTVOy01/GGGP8xo5UjDHG+I0dqRhjjPEbSyrGGGP8xpKKMcYYv7GkYowxxm8sqRhjjPGb/w8+3uBlSlLpaQAAAABJRU5ErkJggg==\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 测试\n",
    "model = sample_transformer\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9)\n",
    "learning_rate = CustomSchedule(optimizer, d_model, warm_steps=4000)\n",
    "\n",
    "lr_list = []\n",
    "for i in range(1, 20000):\n",
    "    learning_rate.step()\n",
    "    lr_list.append(learning_rate.get_lr()[0])\n",
    "plt.figure()\n",
    "plt.plot(np.arange(1, 20000), lr_list)\n",
    "plt.legend(['warmup=4000 steps'])\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.xlabel(\"Train Step\")\n",
    "plt.show()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\n",
      "  \"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\", UserWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/torch/optim/lr_scheduler.py:370: UserWarning: To get the last learning rate computed by the scheduler, please use `get_last_lr()`.\n",
      "  \"please use `get_last_lr()`.\", UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": "<Figure size 432x288 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEGCAYAAABCa2PoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxO0lEQVR4nO3de3ycZZ338c8vk0wyadJTUnoubWl6pOVUCgXdBREpJ+sBtrDiwsraRwFRXB9WVJDloS74KAgrihWQwyMULKgFWVCgKGKhBDlIq4WWFptSekhPSXNOfs8fc0+YppNkSGaSmcz3/XrllZlrrvu6r7tN8+t1NndHRESkt/L6uwIiIjIwKKCIiEhKKKCIiEhKKKCIiEhKKKCIiEhK5Pd3BfpTeXm5T5w4sb+rISKSVV5++eWd7j6iY3pOB5SJEydSWVnZ39UQEckqZvZOonR1eYmISEoooIiISEoooIiISErk9BiKiEBzczNVVVU0NDT0d1UkwxQVFTFu3DgKCgqSyq+AIpLjqqqqKC0tZeLEiZhZf1dHMoS7U11dTVVVFZMmTUrqGnV5ieS4hoYGysrKFEzkAGZGWVnZB2q5KqCIiIKJJPRBfy7U5dUDv3ylio079qesvHmTyvhQRXnKyhMR6Q8KKD3w6GtbWblue0rKcofpo7bxxFf+ISXliWSjJUuWcP/99xMKhcjLy+MnP/kJq1atYvHixRQXF/eozGuvvZaSkhK+9rWvHZAeCoWYPXs2LS0tTJo0ifvuu4+hQ4em4Ckyw8svv8xFF11EfX09Z5xxBrfccstBLY1nn32WhQsXto+NfOpTn+Kaa67p9b0VUHrgrouOTVlZX33wVVZv2pWy8kSyzapVq3jsscf485//TGFhITt37qSpqYlFixZxwQUX9DigdCYSifDqq68CcOGFF3LbbbfxzW9+M6X36E9f/OIX+elPf8pxxx3HGWecwRNPPMHpp59+UL4Pf/jDPPbYYym9t8ZQ+lkkHKK+qbW/qyHSb7Zu3Up5eTmFhYUAlJeXs3z5ct59911OPvlkTj75ZAB++9vfMn/+fI4++mjOPfdcamtrgegWSldeeSWzZ89m3rx5rF+/Pul7z58/ny1bthyU3tbWxiWXXML06dM59dRTOeOMM1i+fDkA1113HcceeyyHH344ixcvJnbq7UknncQVV1zB3LlzmTFjBi+99BKf+tSnqKio4Fvf+hYAmzZtYvr06Vx00UVMnTqVz3zmMzz11FOceOKJVFRUsHr1agBWr17N/PnzOeqoozjhhBNYt25d0n+W+/bt4/jjj8fM+Jd/+Rd+9atfJf3n0VtqofSzSEGIOgUUyRD/+ega1r67L6VlzhwzmG+fPavTzz/2sY9x3XXXMXXqVD760Y+yaNEiLr/8cm666SZWrlxJeXk5O3fu5Prrr+epp55i0KBB3Hjjjdx0003t3TRDhgzhL3/5C/feey9f+cpXkvqfd2trK08//TQXX3wxAJWVldx+++3ccccdPPLII2zatIm1a9eyfft2ZsyYwec+9zkALrvssvb7fvazn+Wxxx7j7LPPBiAcDlNZWcktt9zCwoULefnllxk+fDiHHXYYV1xxBQDr16/nF7/4BXfddRfHHnss999/P3/84x9ZsWIF3/nOd/jVr37F9OnTee6558jPz+epp57iG9/4Bg8//DDr1q1j0aJFCZ/n2WefZcuWLYwbN649bdy4cQkDJkRbhkcccQRjxozhe9/7HrNmdf53lCwFlH5WHA5R39xKW5uTl6eZNpJ7SkpKePnll3nuuedYuXIlixYt4oYbbjggzwsvvMDatWs58cQTAWhqamL+/Pntn59//vnt32O/uDtTX1/PkUceyZYtW5gxYwannnoqAHPnzuWOO+4A4I9//CPnnnsueXl5jBo1qr2VBLBy5Uq++93vUldXx65du5g1a1Z7QPn4xz8OwOzZs5k1axajR48GYPLkyWzevJmhQ4cyadIkZs+eDcCsWbM45ZRTMDNmz57Npk2bANi7dy8XXnghb731FmZGc3MzANOmTWvvruuNo48+mnfeeYeSkhIef/xxPvGJT/DWW2/1uty0BhQzWwDcAoSAO9z9hg6fFwL3AscA1cAid98UfHYVcDHQClzu7k92VaZFR52uB84Nrvmxu9+azudLhUg4+lfQ0NJKcVjxXfpXVy2JdAqFQpx00kmcdNJJzJ49m3vuueeAz92dU089lQceeCDh9fGDzt1NdY2NodTV1XHaaadx2223cfnllydVz4aGBi655BIqKysZP34811577QHrNGLddnl5ee2vY+9bWloOyNMxX3yeq6++mpNPPplf/vKXbNq0iZNOOgmg2xbK2LFjqaqqak+rqqpi7NixB+UdPHhw++szzjiDSy65hJ07d1Je3rvZpmkbQzGzEHAbcDowEzjfzGZ2yHYxsNvdpwA3AzcG184EzgNmAQuAH5lZqJsyLwLGA9PdfQawLF3PlkrF4RCAur0kZ61bt+6A/x2/+uqrHHrooZSWllJTUwPA8ccfz/PPP98+PrJ//37efPPN9msefPDB9u/xLZeuFBcXc+utt/L973+//Rd5zIknnsjDDz9MW1sb27Zt49lnnwVoDx7l5eXU1ta2j6uk2t69e9sDwd13392eHmuhJPoaOnQoo0ePZvDgwbzwwgu4O/feey8LFy48qPz33nuvfexn9erVtLW1UVZW1ut6p/O/xPOA9e7+NoCZLQMWAmvj8iwErg1eLwd+GLQ0FgLL3L0R2Ghm64Py6KLMLwL/7O5tAO6emnm9aRYJAooG5iVX1dbW8qUvfYk9e/aQn5/PlClTWLp0KQ888AALFixgzJgxrFy5krvvvpvzzz+fxsZGAK6//nqmTp0KwO7du5kzZw6FhYUHtGKuv/56fvCDH7S/j//fO8BRRx3FnDlzeOCBB5gxY0b7GMqnP/1pnn76aWbOnMn48eM5+uijGTJkCEOHDuXzn/88hx9+OKNGjeLYY1M34zPelVdeyYUXXsj111/PmWee+YGu/dGPftQ+bfj0009vn+F1++23A/CFL3yB5cuX8+Mf/5j8/HwikQjLli1LzeJWd0/LF3AO0S6p2PvPAj/skOcNYFzc+w1AOfBD4IK49DuD8jotk2iX2TeBSuB/gIpO6rU4yFM5YcIE72+PvrbFD/2Px/xvW/f1d1UkR61du7a/q9Arhx56qO/YsSPl5dbU1Li7+86dO33y5Mm+devWlN8jGyT6+QAqPcHv14HUaV8INLj7XDP7FHAX8OGOmdx9KbAUYO7cud63VTzY+11eLd3kFJG+dNZZZ7Fnzx6ampq4+uqrGTVqVH9XKeOlM6BsITqmETMuSEuUp8rM8oEhRFsaXV3bWXoV8Ejw+pfAz3pZ/z4RKYj+FajLS6RnYjOjUi02biLJS+fCxpeACjObZGZhooPsKzrkWQFcGLw+B3gmaE6tAM4zs0IzmwRUAKu7KfNXQGxu3z8C74/YZbBYC6W+WQFF+o97vzfWJQN90J+LtLVQ3L3FzC4DniQ6xfcud19jZtcR7X9bQXRs5L5g0H0X0QBBkO8hooPtLcCl7t4KkKjM4JY3AD83syuAWuDf0vVsqaRZXtLfioqKqK6u1hb2cgAPzkMpKipK+pq0jqG4++PA4x3Srol73UB03Uiia5cAS5IpM0jfA3yw6RAZQLO8pL+NGzeOqqoqduzY0d9VkQwTO7ExWQNpUD4rxRYzalBe+ktBQUHSJ/KJdEWbQ/az9i4vjaGISJZTQOlnhfl5mKnLS0SynwJKPzMzirXjsIgMAAooGSASVkARkeyngJIBoodsaVBeRLKbAkoGKC7IVwtFRLKeAkoGiASHbImIZDMFlAxQrDEUERkAFFAygAKKiAwECigZIBLO16C8iGQ9BZQMoHUoIjIQKKBkAA3Ki8hAoICSAYrDIW29IiJZTwElAxSHQ7S0OU0tbf1dFRGRHlNAyQCRsI4BFpHsp4CSAd7fwl4zvUQkeymgZAAdAywiA4ECSgaIFOgYYBHJfgooGeD9Y4AVUEQkeymgZIBIOPrXoHPlRSSbKaBkgEiBZnmJSPZTQMkAGpQXkYEgrQHFzBaY2TozW29mX0/weaGZPRh8/qKZTYz77KogfZ2ZndZdmWZ2t5ltNLNXg68j0/lsqfT+tGEFFBHJXvnpKtjMQsBtwKlAFfCSma1w97Vx2S4Gdrv7FDM7D7gRWGRmM4HzgFnAGOApM5saXNNVmf/b3Zen65nSJRKOzfLSGIqIZK90tlDmAevd/W13bwKWAQs75FkI3BO8Xg6cYmYWpC9z90Z33wisD8pLpsyso1leIjIQpDOgjAU2x72vCtIS5nH3FmAvUNbFtd2VucTMXjezm82sMFGlzGyxmVWaWeWOHTs++FOlQSjPCOfnaVBeRLLaQBqUvwqYDhwLDAf+I1Emd1/q7nPdfe6IESP6sn5d0qmNIpLt0hlQtgDj496PC9IS5jGzfGAIUN3FtZ2W6e5bPaoR+BnR7rGsUVygM1FEJLulM6C8BFSY2SQzCxMdZF/RIc8K4MLg9TnAM+7uQfp5wSywSUAFsLqrMs1sdPDdgE8Ab6Tx2VIuojNRRCTLpW2Wl7u3mNllwJNACLjL3deY2XVApbuvAO4E7jOz9cAuogGCIN9DwFqgBbjU3VsBEpUZ3PLnZjYCMOBV4AvperZ0KA7na6W8iGS1tAUUAHd/HHi8Q9o1ca8bgHM7uXYJsCSZMoP0j/S2vv0pojEUEclyA2lQPqsV61x5EclyCigZQrO8RCTbKaBkiEhBvgblRSSrKaBkiEg4T4PyIpLVFFAyRHSWl1ooIpK9FFAyRKQgRGNLG61t3t9VERHpEQWUDBHbwl4zvUQkWymgZIj3D9nSOIqIZCcFlAwRCesYYBHJbgooGULHAItItlNAyRARBRQRyXIKKBmiuCB2DLACiohkJwWUDBE7BlizvEQkWymgZIiIZnmJSJZTQMkQ7etQ1OUlIllKASVDaJaXiGQ7BZQMEdFKeRHJcgooGSIcyiOUZxpDEZGspYCSIcyM4gIdsiUi2UsBJYMUhUMalBeRrKWAkkF0DLCIZLNuA4qZTTWzp83sjeD9HDP7Vvqrlnsi6vISkSyWTAvlp8BVQDOAu78OnJdM4Wa2wMzWmdl6M/t6gs8LzezB4PMXzWxi3GdXBenrzOy0D1DmrWZWm0z9Mk1xOER9swblRSQ7JRNQit19dYe0bn/rmVkIuA04HZgJnG9mMztkuxjY7e5TgJuBG4NrZxINWrOABcCPzCzUXZlmNhcYlsQzZSQdAywi2SyZgLLTzA4DHMDMzgG2JnHdPGC9u7/t7k3AMmBhhzwLgXuC18uBU8zMgvRl7t7o7huB9UF5nZYZBJv/C1yZRN0yUkSD8iKSxfKTyHMpsBSYbmZbgI3AZ5K4biywOe59FXBcZ3ncvcXM9gJlQfoLHa4dG7zurMzLgBXuvjUakxIzs8XAYoAJEyYk8Rh9R4PyIpLNkgko7u4fNbNBQJ6715jZpHRX7IMwszHAucBJ3eV196VEAyRz58719Nbsg1FAEZFslkyX18MA7r7f3WuCtOVJXLcFGB/3flyQljCPmeUDQ4DqLq7tLP0oYAqw3sw2AcVmtj6JOmaUSEE+9VopLyJZqtMWiplNJzooPsTMPhX30WCgKImyXwIqgtbMFqKD7P/cIc8K4EJgFXAO8Iy7u5mtAO43s5uAMUAFsBqwRGW6+xpgVFzda4OB/qwSneXVirvTVbediEgm6qrLaxpwFjAUODsuvQb4fHcFB2MilwFPAiHgLndfY2bXAZXuvgK4E7gvaE3sIpiOHOR7CFhLdEbZpe7eCpCozA/wvBktEg7R5tDY0kZRcIKjiEi26DSguPuvgV+b2Xx3X9WTwt39ceDxDmnXxL1uIDr2kejaJcCSZMpMkKekJ/Xtb/FnoiigiEi2SWZQ/hUzu5Ro91d7V5e7fy5ttcpR7WeiNLdm72IaEclZyQzK30d0fOI04PdEB8JrurxCeiQSO1deA/MikoWSCShT3P1qYL+73wOcycHrSSQFigt0aqOIZK9kAkpz8H2PmR1OdGrvIemrUu6K6BhgEcliyYyhLDWzYcC3iE7zLQGuTmutclQkblBeRCTbdBtQ3P2O4OUfgMkAZpZZe5YMEMVqoYhIFuuyy8vM5pvZOWZ2SPB+jpndDzzfJ7XLMcUF0fiuc+VFJBt1GlDM7P8CdwGfBn5jZtcDvwVeJLpyXVKsvcurWS0UEck+XXV5nQkc5e4NwRjKZuBwd9/UJzXLQeryEpFs1lWXV0Owkh133w28pWCSXhFNGxaRLNZVC2VysEljzKT49+7+8fRVKzfl5RlFBXla2CgiWamrgNLxdMXvp7MiEqVjgEUkW3W1OeTv+7IiEhUp0DHAIpKdklkpL30odiaKiEi2UUDJMDoGWESylQJKhomE1eUlItmp261XzOxRwDsk7wUqgZ/EphZLahSH89leoz9SEck+ybRQ3gZqgZ8GX/uInocyNXgvKRRRl5eIZKlkdhs+wd2PjXv/qJm95O7HmtmAOc89UxRrlpeIZKlkWigl8bsLB69jZ7Y3paVWOUwtFBHJVsm0UP4d+KOZbQAMmARcYmaDgHvSWblcpEF5EclWyZyH8riZVQDTg6R1cQPxP0hXxXJVcUE+Ta1ttLS2kR9K3IC8+/mNbNy5PyX3MzMuOP5QphxS0n1mEZEuJNNCATgGmBjkP8LMcPd701arHNa+43BzK4MTBJSahmaufXQtkYIQhQW9n/W9p66Z/DzjW2fN7HVZIpLbkpk2fB9wGPAqEOuLcaDbgGJmC4BbgBBwh7vf0OHzwqCcY4BqYFFsR2Mzuwq4OLjn5e7+ZFdlmtmdwFyi3XJvAhe5e213dcw08ccADy4qOOjzqt31AHzv3CM4c87oXt/vQzc+Q/V+DYWJSO8l00KZC8x0945rUbpkZiHgNuBUoAp4ycxWuPvauGwXA7vdfYqZnQfcCCwys5nAecAsYAzwlJlNDa7prMwr3H1fcO+bgMuAAwJYNujuTJTNu+oAGD88kpL7lZUUsrO2MSVliUhuS6bP5A1gVA/Kngesd/e33b0JWMbBOxgv5P2B/eXAKWZmQfoyd290943A+qC8TsuMCyYGRDh4MWZWeD+gJN7CfnPQQhk/rDgl9ysfFKa6Vi0UEem9ZAJKObDWzJ40sxWxrySuG0v0lMeYqiAtYR53byG6Ar+si2u7LNPMfga8R3QCwX8nqpSZLTazSjOr3LFjRxKP0bci4WijsbOZXpt31TEoHGJo8cHdYT1RVhJml7q8RCQFkunyujbdlUgVd//XoKvtv4FFwM8S5FkKLAWYO3duxrViuuvyqtpdx/jhxUQbYr03fFAh1fsbcfeUlSkiuSmZacM9PRdlCzA+7v24IC1RniozyweGEB2c7+raLst091YzWwZcSYKAkum6Owa4anc941LU3QVQXhKmudXZ19DCkEhqWj0ikps67fIysz8G32vMbF/cV42Z7Uui7JeACjObZGZhooPsHbvKVgAXBq/PAZ4JBv9XAOeZWaGZTQIqgNWdlWlRU4L6GvBx4G/J/RFkllgLpSHBmSjuzuZddSkbkIdolxdAtQbmRaSXujqx8UPB99KeFOzuLWZ2GfAk0Sm+d7n7GjO7Dqh09xXAncB9ZrYe2EU0QBDkewhYC7QAl7p7K0AnZeYB95jZYKLThl8DvtiTeve34mAMJVELZXddM/ubWlM2IA9QNqgQgOr9TUwekbJiRSQHJbWwMRiXGBmf393/3t117v448HiHtGviXjcA53Zy7RJgSZJltgEndlefbBDpYpZXbMrwuGFqoYhI5klmYeOXgG8D24C2INmBOWmsV84qjlvY2NHm3bE1KKkcQ4m2UHZq6rCI9FIyLZQvA9PcvTrdlREoCOVREDLqEoyhbN4VrEFJYUAZVhxroSigiEjvJLMOZTPR9SHSR4o6OROlancdw4oLKClMdgu27oXz8xgSKWDXfnV5iUjvJPOb6W3gWTP7DdD+W8fdb0pbrXJccTiUeAxld31KWycxZYPC7NTiRhHppWQCyt+Dr3DwJWlWHM5POMuralcd00f3aNJdl8pKwhqUF5Fe6zKgBLO7prr7Z/qoPkJ0cWPHLq+2Nqdqdz2nzhyZ8vuVDSpkw46s25hZRDJMl2MowdqPQ4NFhNJHihMcA7y9ppGm1jbGpaPLqySsLexFpNeSHUN5PtgQsv2YQI2hpE8kHGJfw4FjKFWxKcMpXIMSU1ZSyO66pi5PiRQR6U4yvz02AI8FeUvjviRNisMh6jsMyqdjDUpMeUkY9+hKfBGRnkpmc8j/7IuKyPsSDcrH1qCMHZqGFkqw/cqu/U2MKC1MefkikhuSWSk/gujOvbOAoli6u38kjfXKaZHwwYPym3fVcUhpIUXBbsSpdOD2K2p8ikjPJNPl9XOiO/dOAv4T2ER0119Jk+KCgwflNwfnoKRD2aBoQNFaFBHpjWQCSpm73wk0u/vv3f1zgFonaVQcDlHf3Epb2/vnf23eVZ+WAXmIDsqDNogUkd5JZpZXbKR2q5mdCbwLDE9flSR2DHBjSxuRcIiW1jbe29eQthbK0EgBeab9vESkd5IJKNeb2RDg34kerTsYuCKttcpxxXFb2EfCIbbubaC1zVO6bX28vDxrPwpYRKSnkpnl9Vjwci9wcnqrIxB/JkorZbx/DkoqD9bqqLwkrC3sRaRXuh1DMbOpZva0mb0RvJ9jZt9Kf9VyV/uZKMEW9ulcgxJTVhJmlwblRaQXkhmU/ylwFcFYiru/TnBUr6RHcVwLBaID8qE8Y/SQoq4u65WyQYUalBeRXkkmoBS7++oOaQfvrS4pE1trEtvCvmp3HaOHFKV1W5TojsNqoYhIzyXzG2qnmR1G9NhfzOwcYGtaa5XjioNZXrHFjZt316dtQD6mbFCYmsYWGhKcFCkikoxkAsqlwE+A6Wa2BfgK8IV0VirXHdzlVZfWAXl4fy2KxlFEpKe6DSju/ra7fxQYAUx39w8Bn0x7zXJYJOjyqm9qpaG5le01jWkdkIf3V8ur20tEeirpTnl33+/uNcHbr6apPsKB61Cqdkc3hRw/PM1dXkELZafWoohID/V0lNeSymS2wMzWmdl6M/t6gs8LzezB4PMXzWxi3GdXBenrzOy07so0s58H6W+Y2V1mVtDDZ+t3sTGUuubWuHNQ0ttCKS9RC0VEeqenAcW7yxAcH3wbcDowEzjfzGZ2yHYxsNvdpwA3AzcG184kOjV5FrAA+JGZhbop8+fAdGA2EAH+rYfP1u+KCvIwi3Z5bQ5aKOP6bAxFLRQR6ZlOV8qbWQ2JA4cR/YXdnXnAend/OyhvGbAQWBuXZyFwbfB6OfBDM7MgfZm7NwIbzWx9UB6dlenuj8fVfTUwLok6ZiQzIxLsOFy1q45wfh6HpPmckkHhEIX5eWqhiEiPdRpQ3L23B2OMBTbHva8Cjussj7u3mNleoCxIf6HDtWOD112WGXR1fRb4cqJKmdliYDHAhAkTkn+aPhY7V35vfRPjhkbIy0uql7HHzIzykkJtvyIiPTYQDxD/EfAHd38u0YfuvtTd57r73BEjRvRx1ZIXCY4B3ryrnnFpnuEVM3xQWBtEikiPpTOgbAHGx70fF6QlzGNm+cAQoLqLa7ss08y+TXR6c9bPQisuiB4DvHl3XdoXNcZotbyI9EY6A8pLQIWZTTKzMNFB9hUd8qwALgxenwM84+4epJ8XzAKbBFQAq7sq08z+DTgNON/d29L4XH0iEg6xo7aRPXXNaZ/hFaP9vESkN5I5D6VHgjGRy4AngRBwl7uvMbPrgEp3XwHcCdwXDLrvIth0Msj3ENEB/BbgUndvBUhUZnDL24F3gFXRcX0ecffr0vV86VYcDvGXqr1A+tegxJSXhNm5vwl3J/gzFBFJWtoCCkAw8+rxDmnXxL1uAM7t5NolwJJkygzS0/osfa04HKKmMbo5ZJ+1UErCNLW0sb+plZLCAfXHKSJ9YCAOyg8IsWOAIb3noMQrG6Sz5UWk5xRQMlSkIPpXUxwOMay4bxb9lwWr5TV1WER6QgElQ8W2Xxk/rLjPxjPKS9RCEZGeU0DJULFz5ftqQB6i61AAqrWFvYj0gAJKhioOtrBP9x5e8doDilooItIDCigZ6v0WSt8FlKKCEKWF+RpDEZEe0dzQDBUbQ+mrVfIxZSXhbk9t3L2/icaW1KwdjRSEGNJHkw5EJL0UUDJUbGbX5PJBfXrfspLCLvfzeuHtas5b+kKnn39QZvDbr/wDFSN7uxepiPQ3BZQM9dGZI7n/347r81+0ZYPC/H1XXaefP/O37YRDeVz78Vn0dvLZ3vpmbvifv/HK3/cooIgMAAooGaoglMcJU8r7/L5lJYX8+e97Ov38Txt2ctSEofzzcb3f+r+1zfnBU2/y5raa7jOLSMbToLwcoLwkzK79jbS1HXy22t66Zta8u48TDktNoAvlGRWHlLJOAUVkQFBAkQMMHxSmzWFPffNBn72wsRp3mH9YWcruVzGyRC0UkQFCAUUOUNbFavlVG6opKsjjyPFDU3a/aSNL2bavkb11BwcwEckuCihygPJBne/ntWpDNcdOHE44P3U/NlNHRQfj39yuVopItlNAkQPEWigd16LsqGlk3baalHZ3QbSFArDuPQUUkWyngCIHiO043HEtygtvVwOkbEA+ZvSQIkoL8zWOIjIAKKDIAYYVhzE7uMtr1dvVlBbmc/iYwSm9n5lRMbJELRSRAUABRQ4QyjOGF4cPGpRftaGaeZOGkx9K/Y/MtFGlvLmtBveDpyqLSPZQQJGDDB8UpjquhbJ1bz0bd+5P+fhJzNSRpeyua9amlCJZTgFFDlJWEj5gDGXVhvSMn8RMDQbmNY4ikt0UUOQgZSWFB7RQ/rShmmHFBUwflZ79tqZqppfIgKCAIgcpHxRuP7XR3Vm1oZrjJ5eRl5eeo4jLS8IMHxTmLa1FEclqaQ0oZrbAzNaZ2Xoz+3qCzwvN7MHg8xfNbGLcZ1cF6evM7LTuyjSzy4I0N7O+31VxACkrKWRvfTNNLW1s3lXPlj31nJCm8ROIzvSaqpleIlkvbQHFzELAbcDpwEzgfDOb2SHbxcBud58C3AzcGFw7EzgPmAUsAH5kZqFuynwe+CjwTrqeKVfE1qLsrmviTxt2AqndvyuRqSNLeXNbrWZ6iWSxdLZQ5gHr3f1td28ClgELO+RZCNwTvF4OnGJmFqQvc/dGd98IrA/K67RMd3/F3Tel8XlyRtmg6Gr5nbWN/GlDNSNKCzlsREla7zl1ZCm1jS28u7chrfcRkfRJZ0AZC2yOe18VpCXM4+4twF6grItrkymzS2a22Mwqzaxyx44dH+TSnFFe8v5+XqveruaEw8qw3p6m1Y1pozTTSyTb5dygvLsvdfe57j53xIgR/V2djDQ82CBy9cZqdtQ0Mn9yeru7AKYeEgQUjaOIZK10BpQtwPi49+OCtIR5zCwfGAJUd3FtMmVKL8U2iHz0ta1A+tafxBtSXMDIwYU6bEski6UzoLwEVJjZJDMLEx1kX9EhzwrgwuD1OcAzHh2VXQGcF8wCmwRUAKuTLFN6aXBRPgUh4++76hg7NML44ZE+uW90YF4BRSRbpS2gBGMilwFPAn8FHnL3NWZ2nZl9PMh2J1BmZuuBrwJfD65dAzwErAWeAC5199bOygQws8vNrIpoq+V1M7sjXc820JlZ+8D8/D4YP4mZNrKU9dtraU1w/LCIZL78dBbu7o8Dj3dIuybudQNwbifXLgGWJFNmkH4rcGsvqyyBspIw7+1rSOv6k46mjiqlobmNzbvqmFg+qM/uKyKpkdaAItkrNo6S7vUn8dq3YNlW02lAeW9vA8+9lbrZeZNHlHDMocNSVp5ILlNAkYSmHlLCnromRg/pm/ETgIpDomtd3nyvhtNmjTroc3fn8mWvsHrjrpTdszA/j5evPpWSQv1TEOkt/SuShL5xxgxa+ngsY1BhPuOHR3hze23Cz59fX83qjbu4csE0zp4zptf3++vWfSy+72WeWruNTxz1gZYziUgCCiiSUF6eEU7TZpBdmTayNOFaFHfn+79bx5ghRVz8oUkU5od6fa+xQyOMGVLEo6+9q4AikgI5t7BRMlvFyFI27KilqaXtgPSV67bzyt/38KVTKlISTCAaNM+cM5o/vLWDvXXNKSlTJJcpoEhGmTaylJY2Z1P1/vY0d+em373JhOHFnHPMuJTe7+wjxtDc6jy55r2UliuSixRQJKMkOr3xyTXbeGPLPi4/pYKCFJ9pP3vsECYML+bR199NabkiuUgBRTLK5BGDCOVZ+zhKW5tz8+/eZHL5ID5xZO8H4jsyM84+YjR/2lBNdW1j9xeISKcUUCSjFBWEOLSsuH1Pr8f+spV122r48kcryE9x6yTmrDljaG1z/ucNdXuJ9IYCimScacFhWy2tbfzgqTeZNrI0JdOEOzN9VClTDinh0dfU7SXSGwooknGmjizlner9PFRZxds79nPFqRVpO88eot1eZ80ZzepNu9i2Twd8ifSUAopknGmjSmlzWPKbtcwaMzjhqvlUO2vOGNzhN69vTfu9RAYqBRTJOFNHRrdg2d/UyldPndonux1POaSEGaMH81g3s73cnba21H2JDCRaKS8Z59CyQYTz85g5ejAfmX5In9337CNG890n1lG1u45xw4oP+vzdPfX8689eSukhYBedMJFrPz4rZeWJ9CcFFMk4BaE8fvyZozlsREmfncUCcNbsMXz3iXX85vWt/K9/POyAzzbvquP8n77A3rpmLv/IFEJ5vW/cr9u2j7v/tIn5h5X1SbeeSLopoEhGOmXGyD6/54SyYo4YP5RHX3/3gIDyTvV+zl/6ArWNLfz888cxZ9zQlNyvqaWNd6qf5xuP/IVjDh1GeXBkgEi20hiKSJyz54zmjS372LgzuvXLhh21/NNPVlHf3MoDi49PWTABCOfncdM/HUlNQwvf/OVfiJ5+LZK9FFBE4pw5ZzQAj732Lm9tq2HRT16gtc1Ztng+s8YMSfn9po0q5asfm8qTa7bxy1e2pLx8kb6kgCISZ/SQCMdOHMaDlZs5b+kL5BksW3w800aVpu2en//wZOYeOoxvr1jDu3vq03YfkXTTGIpIB2cfMYZrfr2G0UOKuP/zxzMpzefbh/KM7517BKff8hz/8fDr3Pu5eQdNRmhqaeORP1ex7KXN1De1puS+BfnG//qHwzj7iPTtQiC5RQFFpINPHjWWqt31XHDcoUwoO3j6cDpMLB/EN86cwdW/eoP/98I7fHb+RAAamlt5qHIztz+7gXf3NjBz9OCUBbhN1fv50gOv8Pz6nXz77FlEwqk5Z0ZylwKKSAelRQV844wZfX7fC46bwG/XvMd3Hv8bxxw6nD9t2MnSP7zN9ppG5h46jP/69Bz+oaI8ZVOpm1vbuPl3b/Lj32+g8p3d/PCfj2L6qMEpKVtyk+XyzJK5c+d6ZWVlf1dDpN3WvfV87OY/UNPQAsD8yWV86ZQpzJ9clrY1Oc+v38lXHnyVvfXNXH3WTC44bsIB99pZ28hLG3fx4sZdbNhRS6p+ZZQW5XP67NF8bOZIigrUOsomZvayu889KD2dAcXMFgC3ACHgDne/ocPnhcC9wDFANbDI3TcFn10FXAy0Ape7+5NdlWlmk4BlQBnwMvBZd2/qqn4KKJKJnv7rNh55ZQv/esJE5k4c3if33FnbyNd+8RrPrtvBabNGcsbs0by4cRerN+5i/fZaAIoK8pg2ajD5Kdqoc8vuet7b10BpYT5nzB7NJ48ey7yJwxNuBNrU0sbWvfXUNrak5N4AwweFGTW4qE8Xzw4UfR5QzCwEvAmcClQBLwHnu/vauDyXAHPc/Qtmdh7wSXdfZGYzgQeAecAY4ClganBZwjLN7CHgEXdfZma3A6+5+4+7qqMCisj72tqcu57fyI1P/I3mVqe0MJ+5E4cxb1IZx00ezuFjhhDOT93E0NY258W3q3n4z1t44o2t7G9qZezQCGcfMQazaMDZsqeeqt11bK9pTFnLKN7Q4gKmjyplxujBzBg9mJmjBzOhrJj6plb21Tezr6GFfQ3N7KtvpraxJaX7rw2OFDCitJARJYWUlxQytLig0+DW1uY0t7Wl7N4ABXl5Pd7Fuz8CynzgWnc/LXh/FYC7/1dcnieDPKvMLB94DxgBfD0+byxfcNlBZQI3ADuAUe7e0vHenVFAETnY36vr2NfQzIzRgwml8diAeHVNLfx2zTYe/nMVz6/fSZ4ZY4ZGGDs0wthhEcYNi74eHClIyf3cYXtNA3/duo+/bq1h3Xs11DenZvZcTxWEjLJBhUTCIRqbW2lsaaOppS36vTW1wQTgqa/+I1MOKenRtZ0FlHQOyo8FNse9rwKO6yxPEAj2Eu2yGgu80OHascHrRGWWAXvcvSVB/gOY2WJgMcCECRM+2BOJ5IC+mtkWrziczyeOGssnjhpLXVMLhfmhPgtmEG0tvVO9n79uraFqdx0lRfmUFhUwuCifwZHo99KigpTVqc2dffXNbK9pZGdtEztqGtlZ28iOmkYamlspKghRmJ9HYX6IcH4ehfl5KW0dApQNCqe0PMjBWV7uvhRYCtEWSj9XR0Q6KA73/a+lUJ4xeUQJk0f07H/sPXFIaRFTDknfgtn+kM6V8luA8XHvxwVpCfMEXV5DiA7Od3ZtZ+nVwNCgjM7uJSIiaZTOgPISUGFmk8wsDJwHrOiQZwVwYfD6HOAZjw7qrADOM7PCYPZWBbC6szKDa1YGZRCU+es0PpuIiHSQtrZlMCZyGfAk0Sm+d7n7GjO7Dqh09xXAncB9ZrYe2EU0QBDkewhYC7QAl7p7K0CiMoNb/gewzMyuB14JyhYRkT6ihY2a5SUi8oF0NstLuw2LiEhKKKCIiEhKKKCIiEhKKKCIiEhK5PSgvJntAN7pJls5sLMPqpOJcvnZIbefX8+eu5J5/kPdfUTHxJwOKMkws8pEsxlyQS4/O+T28+vZc/PZoXfPry4vERFJCQUUERFJCQWU7i3t7wr0o1x+dsjt59ez564eP7/GUEREJCXUQhERkZRQQBERkZRQQOmEmS0ws3Vmtt7Mvt7f9Uk3M7vLzLab2RtxacPN7Hdm9lbwfVh/1jFdzGy8ma00s7VmtsbMvhykD/jnN7MiM1ttZq8Fz/6fQfokM3sx+Pl/MDguYsAys5CZvWJmjwXvc+L5zWyTmf3FzF41s8ogrcc/9wooCZhZCLgNOB2YCZxvZjP7t1ZpdzewoEPa14Gn3b0CeDp4PxC1AP/u7jOB44FLg7/vXHj+RuAj7n4EcCSwwMyOB24Ebnb3KcBu4OL+q2Kf+DLw17j3ufT8J7v7kXFrT3r8c6+Aktg8YL27v+3uTcAyYGE/1ymt3P0PRM+kibcQuCd4fQ/wib6sU19x963u/ufgdQ3RXyxjyYHn96ja4G1B8OXAR4DlQfqAfPYYMxsHnAncEbw3cuj5E+jxz70CSmJjgc1x76uCtFwz0t23Bq/fA0b2Z2X6gplNBI4CXiRHnj/o7nkV2A78DtgA7HH3liDLQP/5/wFwJdAWvC8jd57fgd+a2ctmtjhI6/HPfdpObJSBxd3dzAb0HHMzKwEeBr7i7vui/1GNGsjPH5yGeqSZDQV+CUzv3xr1HTM7C9ju7i+b2Un9XJ3+8CF332JmhwC/M7O/xX/4QX/u1UJJbAswPu79uCAt12wzs9EAwfft/VyftDGzAqLB5Ofu/kiQnDPPD+Due4CVwHxgqJnF/sM5kH/+TwQ+bmabiHZtfwS4hRx5fnffEnzfTvQ/E/Poxc+9AkpiLwEVwUyPMNGz7lf0c536wwrgwuD1hcCv+7EuaRP0md8J/NXdb4r7aMA/v5mNCFommFkEOJXoGNJK4Jwg24B8dgB3v8rdx7n7RKL/zp9x98+QA89vZoPMrDT2GvgY8Aa9+LnXSvlOmNkZRPtWQ8Bd7r6kf2uUXmb2AHAS0a2rtwHfBn4FPARMILrN/z+5e8eB+6xnZh8CngP+wvv96N8gOo4yoJ/fzOYQHXgNEf0P5kPufp2ZTSb6P/bhwCvABe7e2H81Tb+gy+tr7n5WLjx/8Iy/DN7mA/e7+xIzK6OHP/cKKCIikhLq8hIRkZRQQBERkZRQQBERkZRQQBERkZRQQBERkZRQQBH5gMysLNid9VUze8/MtsS973JXWjOba2a3fsD7fS7YEfZ1M3vDzBYG6ReZ2ZjePItIKmnasEgvmNm1QK27fy8uLT9uH6jelj8O+D1wtLvvDbaHGeHuG83sWaLrJipTcS+R3lILRSQFzOxuM7vdzF4Evmtm88xsVXDGxp/MbFqQ76S4Mzeuteg5NM+a2dtmdnmCog8BaoBaAHevDYLJOcBc4OdByyhiZseY2e+Djf6ejNs+41kzuyXI94aZzeuLPxPJPQooIqkzDjjB3b8K/A34sLsfBVwDfKeTa6YDpxHdQ+nbwZ5i8V4junPBRjP7mZmdDeDuy4FK4DPufiTRM13+GzjH3Y8B7gLid3coDvJdEnwmknLabVgkdX4R7NwLMAS4x8wqiG4R3jFQxPwm2NKj0cy2E90qvCr2obu3mtkC4FjgFOBmMzvG3a/tUM404HCiO8ZCdCuVrXGfPxCU9wczG2xmQ4PNIEVSRgFFJHX2x73+P8BKd/9kcMbKs51cE78/VCsJ/k16dKBzNbDazH4H/Ay4tkM2A9a4+/xO7tNxsFSDp5Jy6vISSY8hvL/l+UU9LcTMxpjZ0XFJRxLdsA+iYyulwet1wAgzmx9cV2Bms+KuWxSkfwjY6+57e1onkc6ohSKSHt8l2uX1LeA3vSinAPheMD24AdgBfCH47G7gdjOrJ3qGyTnArWY2hOi/7R8Aa4K8DWb2SlDe53pRH5FOadqwyACn6cXSV9TlJSIiKaEWioiIpIRaKCIikhIKKCIikhIKKCIikhIKKCIikhIKKCIikhL/H+7kgl6Y644KAAAAAElFTkSuQmCC\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 其他的学习率调度器测试，例如pytorch自带的StepLR\n",
    "_model = sample_transformer\n",
    "_optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)\n",
    "_learning_rate = torch.optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.5)\n",
    "\n",
    "lr_list = []\n",
    "for i in range(1, 50):\n",
    "    _learning_rate.step()\n",
    "    lr_list.append(_learning_rate.get_lr()[0])\n",
    "plt.figure()\n",
    "plt.plot(np.arange(1, 50), lr_list)\n",
    "plt.legend(['StepLR:gamma=0.5'])\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.xlabel(\"Train Step\")\n",
    "plt.show()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 14.损失和评价准则\n",
    "\n",
    "计算loss时要把mask=1的位置的去除掉\n",
    "\n",
    "### 【大坑！】\n",
    "【注意】，当输入是多维时交叉熵的参数维度，跟tf2不一样，tf2中pred是【b,seq_len,vocab_size】\n",
    "pytorch中pred应该调整为【b,vocab_size,seq_len】"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([3.7972, 1.0699, 0.8568])\n",
      "计算loss时把mask的也算进去了,损失会偏大？： tensor(1.9080)\n",
      "计算loss时去除mask: tensor(1.2441)\n",
      "计算loss时去除mask: tensor(1.2441)\n"
     ]
    }
   ],
   "source": [
    "# 'none'表示直接返回b个样本的loss，默认求平均\n",
    "loss_object = torch.nn.CrossEntropyLoss(reduction='none')\n",
    "# 【注意】，当输入是多维时交叉熵的参数维度，跟tf2不一样，tf2中pred是【b,seq_len,vocab_size】\n",
    "# pytorch中pred应该调整为【b,vocab_size,seq_len】\n",
    "\"\"\"\n",
    "- Input: :math:`(N, C)` where `C = number of classes`, or\n",
    "          :math:`(N, C, d_1, d_2, ..., d_K)` with :math:`K \\geq 1`\n",
    "          in the case of `K`-dimensional loss.\n",
    "\n",
    "- Target: :math:`(N)` where each value is :math:`0 \\leq \\text{targets}[i] \\leq C-1`, or\n",
    "          :math:`(N, d_1, d_2, ..., d_K)` with :math:`K \\geq 1` in the case of\n",
    "          K-dimensional loss.\n",
    "\"\"\"\n",
    "\n",
    "# real [b, targ_seq_len]\n",
    "# pred [b, targ_seq_len, target_vocab_size]\n",
    "def mask_loss_func(real, pred):\n",
    "    # print(real.shape, pred.shape)\n",
    "    # _loss = loss_object(pred, real) # [b, targ_seq_len]\n",
    "    _loss = loss_object(pred.transpose(-1, -2), real)  # [b, targ_seq_len]\n",
    "\n",
    "    # logical_not  取非\n",
    "    # mask 每个元素为bool值，如果real中有pad，则mask相应位置就为False\n",
    "    # mask = torch.logical_not(real.eq(0)).type(_loss.dtype) # [b, targ_seq_len] pad=0的情况\n",
    "    mask = torch.logical_not(real.eq(pad)).type(_loss.dtype)  # [b, targ_seq_len] pad!=0的情况\n",
    "\n",
    "    # 对应位置相乘，token上的损失被保留了下来，pad的loss被置为0或False 去掉，不计算在内\n",
    "    _loss *= mask\n",
    "\n",
    "    return _loss.sum() / mask.sum().item()\n",
    "\n",
    "# 另一种实现方式\n",
    "def mask_loss_func2(real, pred):\n",
    "    # _loss = loss_object(pred, real) # [b, targ_seq_len]\n",
    "    _loss = loss_object(pred.transpose(-1, -2), real)  # [b, targ_seq_len]\n",
    "    # mask = torch.logical_not(real.eq(0)) # [b, targ_seq_len] bool值\n",
    "    mask = torch.logical_not(real.eq(pad)) # [b, targ_seq_len] bool值\n",
    "    _loss = _loss.masked_select(mask) # mask必须是BoolTensor或ByteTensor类型\n",
    "    return _loss.mean()\n",
    "\n",
    "\n",
    "y_pred = torch.randn(3,3) # [3,3]\n",
    "y_true = torch.tensor([1,2,0]) # [3]\n",
    "# print(y_true.shape, y_pred.shape)\n",
    "print(loss_object(y_pred, y_true))\n",
    "print('计算loss时把mask的也算进去了,损失会偏大？：', loss_object(y_pred, y_true).mean())\n",
    "print('计算loss时去除mask:', mask_loss_func(y_true, y_pred))\n",
    "print('计算loss时去除mask:', mask_loss_func2(y_true, y_pred))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "同样计算metric（如acc）时要把mask=1的位置的去除掉"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0, 2, 1])\n",
      "tensor([[-0.3903,  1.7641,  0.5455],\n",
      "        [ 1.0297, -1.5503, -1.5172],\n",
      "        [ 0.8182,  1.1333, -0.4007]])\n",
      "计算acc时去除mask: tensor(0.)\n",
      "计算acc时去除mask: tensor(0.)\n",
      "计算acc时去除mask: tensor(0.)\n"
     ]
    }
   ],
   "source": [
    "# real [b, targ_seq_len]\n",
    "# pred [b, targ_seq_len, target_vocab_size]\n",
    "def mask_accuracy_func(real, pred):\n",
    "    _pred = pred.argmax(dim=-1)  # [b, targ_seq_len, target_vocab_size]=>[b, targ_seq_len]\n",
    "    corrects = _pred.eq(real)  # [b, targ_seq_len] bool值\n",
    "\n",
    "    # logical_not  取非\n",
    "    # mask 每个元素为bool值，如果real中有pad，则mask相应位置就为False\n",
    "    # mask = torch.logical_not(real.eq(0)) # [b, targ_seq_len] bool值 pad=0的情况\n",
    "    mask = torch.logical_not(real.eq(pad))  # [b, targ_seq_len] bool值 pad!=0的情况\n",
    "\n",
    "    # 对应位置相乘，token上的值被保留了下来，pad上的值被置为0或False 去掉，不计算在内\n",
    "    corrects *= mask\n",
    "\n",
    "    return corrects.sum().float() / mask.sum().item()\n",
    "\n",
    "# 另一种实现方式\n",
    "def mask_accuracy_func2(real, pred):\n",
    "    _pred = pred.argmax(dim=-1) # [b, targ_seq_len, target_vocab_size]=>[b, targ_seq_len]\n",
    "    corrects = _pred.eq(real).type(torch.float32) # [b, targ_seq_len]\n",
    "    # mask = torch.logical_not(real.eq(0)) # [b, targ_seq_len] bool值\n",
    "    mask = torch.logical_not(real.eq(pad)) # [b, targ_seq_len] bool值\n",
    "    corrects = corrects.masked_select(mask) # [真正有token的个数] 平摊开成1维的\n",
    "\n",
    "    return corrects.mean()\n",
    "\n",
    "def mask_accuracy_func3(real, pred):\n",
    "    _pred = pred.argmax(dim=-1) # [b, targ_seq_len, target_vocab_size]=>[b, targ_seq_len]\n",
    "    corrects = _pred.eq(real) # [b, targ_seq_len] bool值\n",
    "    # mask = torch.logical_not(real.eq(0)) # [b, targ_seq_len] bool值\n",
    "    mask = torch.logical_not(real.eq(pad)) # [b, targ_seq_len] bool值\n",
    "    corrects = torch.logical_and(corrects, mask)\n",
    "    # print(corrects.dtype) # bool\n",
    "    # print(corrects.sum().dtype) #int64\n",
    "    return corrects.sum().float()/mask.sum().item()\n",
    "\n",
    "y_pred = torch.randn(3,3) # [3,3]\n",
    "y_true = torch.tensor([0,2,1]) # [3] 最后一个1表示pad噢~\n",
    "print(y_true)\n",
    "print(y_pred)\n",
    "print('计算acc时去除mask:', mask_accuracy_func(y_true, y_pred))\n",
    "print('计算acc时去除mask:', mask_accuracy_func2(y_true, y_pred))\n",
    "print('计算acc时去除mask:', mask_accuracy_func3(y_true, y_pred))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 15.生成mask\n",
    "\n",
    "$$Attention(Q,K,V)=softmax_{(k)}(\\frac{QK^T}{\\sqrt{d_k}})V$$\n",
    "\n",
    "注意：实现时对mask的处理\n",
    "\n",
    "mask=1的位置是pad或者future token，乘以-1e9（-1*10^9）成为负无穷，经过softmax后会趋于0\n",
    "\n",
    "【注意】：在计算decoder的第2个attention block时（encoder-decoder attention），\n",
    "从scaled dot-product的公式可以看出，此时Q是decoder的第一个attention block的输出，\n",
    "而K，V都来自encoder的输出。QK^T后得到[...,seq_len_q, seq_len_k]，而softmax是在seq_len_k维\n",
    "上进行的，softmax后seq_len_k维上pad的位置被置于0，乘以V（seq_len_k=seq_len_v=encoder output的seq_len）\n",
    "后那些pad的位置还是只会是0。所以仅从公式就可以看出decoder的第2个attention block的padding mask是基于\n",
    "encoder output 的seq_len即整个模型的inp_seq_len来设置的，而不是targ_seq_len。\n",
    "\n",
    "从注意力角度来看，可以理解成：当 Q 接收到decoder的第一个注意力块的输出，并且 K 接收到encoder的输出时，注意力权重表示根据encoder的输出赋予decoder输入的重要性。\n",
    "换一种说法，decoder通过查看encoder输出和对其自身输出的自注意`力，预测下一个词。\n",
    "\n",
    "另外 padding mask 的维度是[b,1,1,seq_len] 一般只关注最后一个seq_len维度上pad(无论是在\n",
    "self-attention 或者encoder-decoder attention上都比较容易理解)\n",
    "\n",
    "而 look-ahead mask 是decoder self-attention时用于mask future token的，因为是“自”注意力，\n",
    "所以维度是[...,seq_len, seq_len]，需要自己跟自己运算，所以在QK（此时Q=K=V）矩阵乘的结果的最后2个维度上都需要mask"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "outputs": [],
   "source": [
    "# inp [b, inp_seq_len] 序列已经加入pad填充\n",
    "# targ [b, targ_seq_len] 序列已经加入pad填充\n",
    "def create_mask(inp, targ):\n",
    "    # encoder padding mask\n",
    "    enc_padding_mask = create_padding_mask(inp)  # =>[b,1,1,inp_seq_len] mask=1的位置为pad\n",
    "\n",
    "    # decoder's first attention block(self-attention)\n",
    "    # 使用的padding create_mask & look-ahead create_mask\n",
    "    look_ahead_mask = create_look_ahead_mask(targ.shape[-1])  # =>[targ_seq_len,targ_seq_len] ##################\n",
    "    dec_targ_padding_mask = create_padding_mask(targ)  # =>[b,1,1,targ_seq_len]\n",
    "    combined_mask = torch.max(look_ahead_mask, dec_targ_padding_mask)  # 结合了2种mask =>[b,1,targ_seq_len,targ_seq_len]\n",
    "\n",
    "    # decoder's second attention block(encoder-decoder attention) 使用的padding create_mask\n",
    "    # 【注意】：这里的mask是用于遮挡encoder output的填充pad，而encoder的输出与其输入shape都是[b,inp_seq_len,d_model]\n",
    "    # 所以这里mask的长度是inp_seq_len而不是targ_mask_len\n",
    "    dec_padding_mask = create_padding_mask(inp)  # =>[b,1,1,inp_seq_len] mask=1的位置为pad\n",
    "\n",
    "    return enc_padding_mask, combined_mask, dec_padding_mask\n",
    "    # [b,1,1,inp_seq_len], [b,1,targ_seq_len,targ_seq_len], [b,1,1,inp_seq_len]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 16.训练和保存\n",
    "\n",
    "### 配置检查点 save_dir"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Transformer(\n",
      "  (encoder): Encoder(\n",
      "    (embedding): Embedding(3901, 128)\n",
      "    (enc_layers): ModuleList(\n",
      "      (0): EncoderLayer(\n",
      "        (mha): MultiHeadAttention(\n",
      "          (wq): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wk): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wv): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (final_linear): Linear(in_features=128, out_features=128, bias=True)\n",
      "        )\n",
      "        (ffn): Sequential(\n",
      "          (0): Linear(in_features=128, out_features=512, bias=True)\n",
      "          (1): ReLU()\n",
      "          (2): Linear(in_features=512, out_features=128, bias=True)\n",
      "        )\n",
      "        (layernorm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (layernorm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (dropout1): Dropout(p=0.1, inplace=False)\n",
      "        (dropout2): Dropout(p=0.1, inplace=False)\n",
      "      )\n",
      "      (1): EncoderLayer(\n",
      "        (mha): MultiHeadAttention(\n",
      "          (wq): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wk): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wv): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (final_linear): Linear(in_features=128, out_features=128, bias=True)\n",
      "        )\n",
      "        (ffn): Sequential(\n",
      "          (0): Linear(in_features=128, out_features=512, bias=True)\n",
      "          (1): ReLU()\n",
      "          (2): Linear(in_features=512, out_features=128, bias=True)\n",
      "        )\n",
      "        (layernorm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (layernorm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (dropout1): Dropout(p=0.1, inplace=False)\n",
      "        (dropout2): Dropout(p=0.1, inplace=False)\n",
      "      )\n",
      "      (2): EncoderLayer(\n",
      "        (mha): MultiHeadAttention(\n",
      "          (wq): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wk): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wv): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (final_linear): Linear(in_features=128, out_features=128, bias=True)\n",
      "        )\n",
      "        (ffn): Sequential(\n",
      "          (0): Linear(in_features=128, out_features=512, bias=True)\n",
      "          (1): ReLU()\n",
      "          (2): Linear(in_features=512, out_features=128, bias=True)\n",
      "        )\n",
      "        (layernorm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (layernorm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (dropout1): Dropout(p=0.1, inplace=False)\n",
      "        (dropout2): Dropout(p=0.1, inplace=False)\n",
      "      )\n",
      "      (3): EncoderLayer(\n",
      "        (mha): MultiHeadAttention(\n",
      "          (wq): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wk): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wv): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (final_linear): Linear(in_features=128, out_features=128, bias=True)\n",
      "        )\n",
      "        (ffn): Sequential(\n",
      "          (0): Linear(in_features=128, out_features=512, bias=True)\n",
      "          (1): ReLU()\n",
      "          (2): Linear(in_features=512, out_features=128, bias=True)\n",
      "        )\n",
      "        (layernorm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (layernorm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (dropout1): Dropout(p=0.1, inplace=False)\n",
      "        (dropout2): Dropout(p=0.1, inplace=False)\n",
      "      )\n",
      "    )\n",
      "    (dropout): Dropout(p=0.1, inplace=False)\n",
      "  )\n",
      "  (decoder): Decoder(\n",
      "    (embedding): Embedding(2591, 128)\n",
      "    (dec_layers): ModuleList(\n",
      "      (0): DecoderLayer(\n",
      "        (mha1): MultiHeadAttention(\n",
      "          (wq): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wk): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wv): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (final_linear): Linear(in_features=128, out_features=128, bias=True)\n",
      "        )\n",
      "        (mha2): MultiHeadAttention(\n",
      "          (wq): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wk): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wv): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (final_linear): Linear(in_features=128, out_features=128, bias=True)\n",
      "        )\n",
      "        (ffn): Sequential(\n",
      "          (0): Linear(in_features=128, out_features=512, bias=True)\n",
      "          (1): ReLU()\n",
      "          (2): Linear(in_features=512, out_features=128, bias=True)\n",
      "        )\n",
      "        (layernorm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (layernorm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (layernorm3): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (dropout1): Dropout(p=0.1, inplace=False)\n",
      "        (dropout2): Dropout(p=0.1, inplace=False)\n",
      "        (dropout3): Dropout(p=0.1, inplace=False)\n",
      "      )\n",
      "      (1): DecoderLayer(\n",
      "        (mha1): MultiHeadAttention(\n",
      "          (wq): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wk): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wv): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (final_linear): Linear(in_features=128, out_features=128, bias=True)\n",
      "        )\n",
      "        (mha2): MultiHeadAttention(\n",
      "          (wq): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wk): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wv): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (final_linear): Linear(in_features=128, out_features=128, bias=True)\n",
      "        )\n",
      "        (ffn): Sequential(\n",
      "          (0): Linear(in_features=128, out_features=512, bias=True)\n",
      "          (1): ReLU()\n",
      "          (2): Linear(in_features=512, out_features=128, bias=True)\n",
      "        )\n",
      "        (layernorm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (layernorm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (layernorm3): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (dropout1): Dropout(p=0.1, inplace=False)\n",
      "        (dropout2): Dropout(p=0.1, inplace=False)\n",
      "        (dropout3): Dropout(p=0.1, inplace=False)\n",
      "      )\n",
      "      (2): DecoderLayer(\n",
      "        (mha1): MultiHeadAttention(\n",
      "          (wq): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wk): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wv): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (final_linear): Linear(in_features=128, out_features=128, bias=True)\n",
      "        )\n",
      "        (mha2): MultiHeadAttention(\n",
      "          (wq): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wk): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wv): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (final_linear): Linear(in_features=128, out_features=128, bias=True)\n",
      "        )\n",
      "        (ffn): Sequential(\n",
      "          (0): Linear(in_features=128, out_features=512, bias=True)\n",
      "          (1): ReLU()\n",
      "          (2): Linear(in_features=512, out_features=128, bias=True)\n",
      "        )\n",
      "        (layernorm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (layernorm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (layernorm3): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (dropout1): Dropout(p=0.1, inplace=False)\n",
      "        (dropout2): Dropout(p=0.1, inplace=False)\n",
      "        (dropout3): Dropout(p=0.1, inplace=False)\n",
      "      )\n",
      "      (3): DecoderLayer(\n",
      "        (mha1): MultiHeadAttention(\n",
      "          (wq): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wk): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wv): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (final_linear): Linear(in_features=128, out_features=128, bias=True)\n",
      "        )\n",
      "        (mha2): MultiHeadAttention(\n",
      "          (wq): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wk): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (wv): Linear(in_features=128, out_features=128, bias=True)\n",
      "          (final_linear): Linear(in_features=128, out_features=128, bias=True)\n",
      "        )\n",
      "        (ffn): Sequential(\n",
      "          (0): Linear(in_features=128, out_features=512, bias=True)\n",
      "          (1): ReLU()\n",
      "          (2): Linear(in_features=512, out_features=128, bias=True)\n",
      "        )\n",
      "        (layernorm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (layernorm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (layernorm3): LayerNorm((128,), eps=1e-06, elementwise_affine=True)\n",
      "        (dropout1): Dropout(p=0.1, inplace=False)\n",
      "        (dropout2): Dropout(p=0.1, inplace=False)\n",
      "        (dropout3): Dropout(p=0.1, inplace=False)\n",
      "      )\n",
      "    )\n",
      "    (dropout): Dropout(p=0.1, inplace=False)\n",
      "  )\n",
      "  (final_layer): Linear(in_features=128, out_features=2591, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "save_dir = cwd + './save/'\n",
    "\n",
    "transformer = Transformer(num_layers,\n",
    "                          d_model,\n",
    "                          num_heads,\n",
    "                          dff,\n",
    "                          input_vocab_size,\n",
    "                          target_vocab_size,\n",
    "                          pe_input=input_vocab_size,\n",
    "                          pe_target=target_vocab_size,\n",
    "                          rate=dropout_rate)\n",
    "\n",
    "print(transformer) # 打印模型基本信息\n",
    "\n",
    "transformer = transformer.to(device)\n",
    "if ngpu > 1: # 并行化\n",
    "    transformer = torch.nn.DataParallel(transformer,  device_ids=list(range(ngpu))) # 设置并行执行  device_ids=[0,1]\n",
    "\n",
    "optimizer = torch.optim.Adam(transformer.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9)\n",
    "lr_scheduler = CustomSchedule(optimizer, d_model, warm_steps=4000)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "source": [
    "# 关于position encoding的一点自己的理解：\n",
    "\"\"\"\n",
    "为什么这里传入的pe_input是input_vocab_size，也就是说position encoding传入的pos参数是词表大小\n",
    "而我一开始理解的max_seq_len的大小，所以才在encoder中获取位置编码是self.pos_encoding[:, :seq_len, :]\n",
    "但知道看到这里传入的是词表大小我才发现我理解错了\n",
    "\n",
    "其实position encoding 就跟embedding 一样，就是【vocab_size x d_model】的\n",
    "但此时另一个问题又出现了，那为什么在获取position encoding时 使用的是pos_encoding[:, :seq_len, :]而不是\n",
    "像embedding一样 去look-up 查询每个token的位置表示呢？而是直接取得前seq_len个表示作为位置表示？\n",
    "\n",
    "这就体现了这种位置编码的巧妙之处，他不需要训练，就能独特的表示每一个单独的token，而每个token之间又存在关系\n",
    "所以这并不需要位置编码和token是一一对应（固定死的），只需要位置编码能传达这2点信息（即token的独特性和相对依赖性）就够了\n",
    "即使是同一个token 在一个句子的不同的位置时 他的位置编码当然也是不一样的了 如果像embedding那样去查 那一个句子的\n",
    "不同位置的相同token  其表示就会是一样的  就体现不出位置关系\n",
    "\n",
    "这样一想的话，似乎把pe_input设置成max_seq_len也是可以的。但是这里实现position encoding方式（sin  cos）\n",
    "可以在测试阶段接受长度超过训练集实例的情况！所以干脆把pe_input设置成词表大小？\n",
    "\"\"\"\n",
    "\n",
    "# inp [b,inp_seq_len]\n",
    "# targ [b,targ_seq_len]\n",
    "\"\"\"\n",
    "拆分targ, 例如：sentence = \"SOS A lion in the jungle is sleeping EOS\"\n",
    "tar_inp = \"<start>> A lion in the jungle is sleeping\"\n",
    "tar_real = \"A lion in the jungle is sleeping <end>\"\n",
    "\"\"\"\n",
    "def train_step(model, inp, targ):\n",
    "    # 目标（target）被分成了 tar_inp 和 tar_real\n",
    "    # tar_inp 作为输入传递到解码器。\n",
    "    # tar_real 是位移了 1 的同一个输入：在 tar_inp 中的每个位置，tar_real 包含了应该被预测到的下一个标记（token）。\n",
    "    targ_inp = targ[:, :-1]\n",
    "    targ_real = targ[:, 1:]\n",
    "\n",
    "    enc_padding_mask, combined_mask, dec_padding_mask = create_mask(inp, targ_inp)\n",
    "\n",
    "    inp = inp.to(device)\n",
    "    targ_inp = targ_inp.to(device)\n",
    "    targ_real = targ_real.to(device)\n",
    "    enc_padding_mask = enc_padding_mask.to(device)\n",
    "    combined_mask = combined_mask.to(device)\n",
    "    dec_padding_mask = dec_padding_mask.to(device)\n",
    "    # print('device:', inp.device, targ_inp)\n",
    "\n",
    "    model.train()  # 设置train mode\n",
    "\n",
    "    optimizer.zero_grad()  # 梯度清零\n",
    "\n",
    "    # forward\n",
    "    prediction, _ = transformer(inp, targ_inp, enc_padding_mask, combined_mask, dec_padding_mask)\n",
    "    # [b, targ_seq_len, target_vocab_size]\n",
    "    # {'..block1': [b, num_heads, targ_seq_len, targ_seq_len],\n",
    "    #  '..block2': [b, num_heads, targ_seq_len, inp_seq_len], ...}\n",
    "\n",
    "    loss = mask_loss_func(targ_real, prediction)\n",
    "    metric = mask_accuracy_func(targ_real, prediction)\n",
    "\n",
    "    # backward\n",
    "    loss.backward()  # 反向传播计算梯度\n",
    "    optimizer.step()  # 更新参数\n",
    "\n",
    "    return loss.item(), metric.item()\n",
    "\n",
    "\n",
    "# 检查train_step()的效果\n",
    "batch_src, batch_targ = next(iter(train_dataloader)) # [64,10], [64,10]\n",
    "print(train_step(transformer, batch_src, batch_targ))\n",
    "\"\"\"\n",
    "x += pos_encoding  # [b, inp_seq_len, d_model]\n",
    "RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!\n",
    "\"\"\""
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "execution_count": 29,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8.07600212097168, 0.0)\n"
     ]
    },
    {
     "data": {
      "text/plain": "'\\nx += pos_encoding  # [b, inp_seq_len, d_model]\\nRuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!\\n'"
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "outputs": [],
   "source": [
    "def validate_step(model, inp, targ):\n",
    "    targ_inp = targ[:, :-1]\n",
    "    targ_real = targ[:, 1:]\n",
    "\n",
    "    enc_padding_mask, combined_mask, dec_padding_mask = create_mask(inp, targ_inp)\n",
    "\n",
    "    inp = inp.to(device)\n",
    "    targ_inp = targ_inp.to(device)\n",
    "    targ_real = targ_real.to(device)\n",
    "    enc_padding_mask = enc_padding_mask.to(device)\n",
    "    combined_mask = combined_mask.to(device)\n",
    "    dec_padding_mask = dec_padding_mask.to(device)\n",
    "\n",
    "    model.eval()  # 设置eval mode\n",
    "\n",
    "    with torch.no_grad():\n",
    "        # forward\n",
    "        prediction, _ = model(inp, targ_inp, enc_padding_mask, combined_mask, dec_padding_mask)\n",
    "        # [b, targ_seq_len, target_vocab_size]\n",
    "        # {'..block1': [b, num_heads, targ_seq_len, targ_seq_len],\n",
    "        #  '..block2': [b, num_heads, targ_seq_len, inp_seq_len], ...}\n",
    "\n",
    "        val_loss = mask_loss_func(targ_real, prediction)\n",
    "        val_metric = mask_accuracy_func(targ_real, prediction)\n",
    "\n",
    "    return val_loss.item(), val_metric.item()\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "outputs": [],
   "source": [
    "EPOCHS = 40 # 50 # 30  # 20\n",
    "\n",
    "print_trainstep_every = 50  # 每50个step做一次打印\n",
    "\n",
    "metric_name = 'acc'\n",
    "# df_history = pd.DataFrame(columns=['epoch', 'loss', metric_name]) # 记录训练历史信息\n",
    "df_history = pd.DataFrame(columns=['epoch', 'loss', metric_name, 'val_loss', 'val_' + metric_name])\n",
    "\n",
    "\n",
    "# 打印时间\n",
    "def printbar():\n",
    "    nowtime = datetime.datetime.now().strftime('%Y-%m_%d %H:%M:%S')\n",
    "    print('\\n' + \"==========\"*8 + '%s'%nowtime)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "outputs": [],
   "source": [
    "def train_model(model, epochs, train_dataloader, val_dataloader, print_every):\n",
    "    starttime = time.time()\n",
    "    print('*' * 27, 'start training...')\n",
    "    printbar()\n",
    "\n",
    "    best_acc = 0.\n",
    "    for epoch in range(1, epochs + 1):\n",
    "\n",
    "        # lr_scheduler.step() # 更新学习率\n",
    "\n",
    "        loss_sum = 0.\n",
    "        metric_sum = 0.\n",
    "\n",
    "        for step, (inp, targ) in enumerate(train_dataloader, start=1):\n",
    "            # inp [64, 10] , targ [64, 10]\n",
    "            loss, metric = train_step(model, inp, targ)\n",
    "\n",
    "            loss_sum += loss\n",
    "            metric_sum += metric\n",
    "\n",
    "            # 打印batch级别日志\n",
    "            if step % print_every == 0:\n",
    "                print('*' * 8, f'[step = {step}] loss: {loss_sum / step:.3f}, {metric_name}: {metric_sum / step:.3f}')\n",
    "\n",
    "            lr_scheduler.step()  # 更新学习率\n",
    "\n",
    "        # 一个epoch的train结束，做一次验证\n",
    "        # test(model, train_dataloader)\n",
    "        val_loss_sum = 0.\n",
    "        val_metric_sum = 0.\n",
    "        for val_step, (inp, targ) in enumerate(val_dataloader, start=1):\n",
    "            # inp [64, 10] , targ [64, 10]\n",
    "            loss, metric = validate_step(model, inp, targ)\n",
    "\n",
    "            val_loss_sum += loss\n",
    "            val_metric_sum += metric\n",
    "\n",
    "        # 记录和收集1个epoch的训练（和验证）信息\n",
    "        # record = (epoch, loss_sum/step, metric_sum/step)\n",
    "        record = (epoch, loss_sum/step, metric_sum/step, val_loss_sum/val_step, val_metric_sum/val_step)\n",
    "        df_history.loc[epoch - 1] = record\n",
    "\n",
    "        # 打印epoch级别的日志\n",
    "        # print('*'*8, 'EPOCH = {} loss: {:.3f}, {}: {:.3f}'.format(\n",
    "        #        record[0], record[1], metric_name, record[2]))\n",
    "        print('EPOCH = {} loss: {:.3f}, {}: {:.3f}, val_loss: {:.3f}, val_{}: {:.3f}'.format(\n",
    "            record[0], record[1], metric_name, record[2], record[3], metric_name, record[4]))\n",
    "        printbar()\n",
    "\n",
    "        # 保存模型\n",
    "        # current_acc_avg = metric_sum / step\n",
    "        current_acc_avg = val_metric_sum / val_step # 看验证集指标\n",
    "        if current_acc_avg > best_acc:  # 保存更好的模型\n",
    "            best_acc = current_acc_avg\n",
    "            checkpoint = save_dir + '{:03d}_{:.2f}_ckpt.tar'.format(epoch, current_acc_avg)\n",
    "            if device.type == 'cuda' and ngpu > 1:\n",
    "                # model_sd = model.module.state_dict()  ##################\n",
    "                model_sd = copy.deepcopy(model.module.state_dict())\n",
    "            else:\n",
    "                # model_sd = model.state_dict(),  ##################\n",
    "                model_sd = copy.deepcopy(model.state_dict())  ##################\n",
    "            torch.save({\n",
    "                'loss': loss_sum / step,\n",
    "                'epoch': epoch,\n",
    "                'net': model_sd,\n",
    "                'opt': optimizer.state_dict(),\n",
    "                'lr_scheduler': lr_scheduler.state_dict()\n",
    "            }, checkpoint)\n",
    "\n",
    "\n",
    "    print('finishing training...')\n",
    "    endtime = time.time()\n",
    "    time_elapsed = endtime - starttime\n",
    "    print('Training complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))\n",
    "    return df_history"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*************************** start training...\n",
      "\n",
      "================================================================================2020-12_15 12:54:02\n",
      "EPOCH = 1 loss: 7.786, acc: 0.006, val_loss: 7.445, val_acc: 0.119\n",
      "\n",
      "================================================================================2020-12_15 12:54:08\n",
      "EPOCH = 2 loss: 7.084, acc: 0.191, val_loss: 6.532, val_acc: 0.292\n",
      "\n",
      "================================================================================2020-12_15 12:54:15\n",
      "EPOCH = 3 loss: 6.306, acc: 0.293, val_loss: 5.850, val_acc: 0.336\n",
      "\n",
      "================================================================================2020-12_15 12:54:21\n",
      "EPOCH = 4 loss: 5.689, acc: 0.359, val_loss: 5.239, val_acc: 0.428\n",
      "\n",
      "================================================================================2020-12_15 12:54:26\n",
      "EPOCH = 5 loss: 5.135, acc: 0.415, val_loss: 4.753, val_acc: 0.422\n",
      "\n",
      "================================================================================2020-12_15 12:54:33\n",
      "EPOCH = 6 loss: 4.667, acc: 0.440, val_loss: 4.312, val_acc: 0.470\n",
      "\n",
      "================================================================================2020-12_15 12:54:38\n",
      "EPOCH = 7 loss: 4.234, acc: 0.485, val_loss: 3.898, val_acc: 0.529\n",
      "\n",
      "================================================================================2020-12_15 12:54:44\n",
      "EPOCH = 8 loss: 3.836, acc: 0.524, val_loss: 3.562, val_acc: 0.543\n",
      "\n",
      "================================================================================2020-12_15 12:54:49\n",
      "EPOCH = 9 loss: 3.514, acc: 0.544, val_loss: 3.306, val_acc: 0.566\n",
      "\n",
      "================================================================================2020-12_15 12:54:55\n",
      "EPOCH = 10 loss: 3.273, acc: 0.557, val_loss: 3.105, val_acc: 0.574\n",
      "\n",
      "================================================================================2020-12_15 12:55:01\n",
      "EPOCH = 11 loss: 3.072, acc: 0.568, val_loss: 2.942, val_acc: 0.581\n",
      "\n",
      "================================================================================2020-12_15 12:55:07\n",
      "EPOCH = 12 loss: 2.901, acc: 0.576, val_loss: 2.805, val_acc: 0.590\n",
      "\n",
      "================================================================================2020-12_15 12:55:14\n",
      "EPOCH = 13 loss: 2.767, acc: 0.587, val_loss: 2.683, val_acc: 0.604\n",
      "\n",
      "================================================================================2020-12_15 12:55:20\n",
      "EPOCH = 14 loss: 2.633, acc: 0.600, val_loss: 2.573, val_acc: 0.614\n",
      "\n",
      "================================================================================2020-12_15 12:55:27\n",
      "EPOCH = 15 loss: 2.517, acc: 0.610, val_loss: 2.482, val_acc: 0.620\n",
      "\n",
      "================================================================================2020-12_15 12:55:33\n",
      "EPOCH = 16 loss: 2.418, acc: 0.619, val_loss: 2.400, val_acc: 0.628\n",
      "\n",
      "================================================================================2020-12_15 12:55:39\n",
      "EPOCH = 17 loss: 2.324, acc: 0.628, val_loss: 2.318, val_acc: 0.637\n",
      "\n",
      "================================================================================2020-12_15 12:55:45\n",
      "EPOCH = 18 loss: 2.237, acc: 0.637, val_loss: 2.247, val_acc: 0.644\n",
      "\n",
      "================================================================================2020-12_15 12:55:50\n",
      "EPOCH = 19 loss: 2.159, acc: 0.644, val_loss: 2.179, val_acc: 0.652\n",
      "\n",
      "================================================================================2020-12_15 12:55:56\n",
      "EPOCH = 20 loss: 2.080, acc: 0.653, val_loss: 2.120, val_acc: 0.659\n",
      "\n",
      "================================================================================2020-12_15 12:56:02\n",
      "EPOCH = 21 loss: 2.008, acc: 0.659, val_loss: 2.094, val_acc: 0.658\n",
      "\n",
      "================================================================================2020-12_15 12:56:08\n",
      "EPOCH = 22 loss: 1.937, acc: 0.667, val_loss: 2.003, val_acc: 0.670\n",
      "\n",
      "================================================================================2020-12_15 12:56:14\n",
      "EPOCH = 23 loss: 1.862, acc: 0.675, val_loss: 1.949, val_acc: 0.677\n",
      "\n",
      "================================================================================2020-12_15 12:56:20\n",
      "EPOCH = 24 loss: 1.804, acc: 0.680, val_loss: 1.904, val_acc: 0.683\n",
      "\n",
      "================================================================================2020-12_15 12:56:26\n",
      "EPOCH = 25 loss: 1.744, acc: 0.687, val_loss: 1.863, val_acc: 0.687\n",
      "\n",
      "================================================================================2020-12_15 12:56:32\n",
      "EPOCH = 26 loss: 1.664, acc: 0.698, val_loss: 1.811, val_acc: 0.692\n",
      "\n",
      "================================================================================2020-12_15 12:56:39\n",
      "EPOCH = 27 loss: 1.610, acc: 0.703, val_loss: 1.767, val_acc: 0.700\n",
      "\n",
      "================================================================================2020-12_15 12:56:46\n",
      "EPOCH = 28 loss: 1.550, acc: 0.709, val_loss: 1.719, val_acc: 0.708\n",
      "\n",
      "================================================================================2020-12_15 12:56:52\n",
      "EPOCH = 29 loss: 1.485, acc: 0.719, val_loss: 1.701, val_acc: 0.711\n",
      "\n",
      "================================================================================2020-12_15 12:56:58\n",
      "EPOCH = 30 loss: 1.429, acc: 0.726, val_loss: 1.667, val_acc: 0.714\n",
      "\n",
      "================================================================================2020-12_15 12:57:04\n",
      "EPOCH = 31 loss: 1.373, acc: 0.733, val_loss: 1.611, val_acc: 0.722\n",
      "\n",
      "================================================================================2020-12_15 12:57:10\n",
      "EPOCH = 32 loss: 1.316, acc: 0.742, val_loss: 1.613, val_acc: 0.720\n",
      "\n",
      "================================================================================2020-12_15 12:57:17\n",
      "EPOCH = 33 loss: 1.262, acc: 0.749, val_loss: 1.543, val_acc: 0.733\n",
      "\n",
      "================================================================================2020-12_15 12:57:22\n",
      "EPOCH = 34 loss: 1.211, acc: 0.756, val_loss: 1.511, val_acc: 0.736\n",
      "\n",
      "================================================================================2020-12_15 12:57:28\n",
      "EPOCH = 35 loss: 1.151, acc: 0.766, val_loss: 1.484, val_acc: 0.743\n",
      "\n",
      "================================================================================2020-12_15 12:57:34\n",
      "EPOCH = 36 loss: 1.099, acc: 0.772, val_loss: 1.481, val_acc: 0.741\n",
      "\n",
      "================================================================================2020-12_15 12:57:41\n",
      "EPOCH = 37 loss: 1.048, acc: 0.781, val_loss: 1.435, val_acc: 0.750\n",
      "\n",
      "================================================================================2020-12_15 12:57:47\n",
      "EPOCH = 38 loss: 1.000, acc: 0.788, val_loss: 1.402, val_acc: 0.756\n",
      "\n",
      "================================================================================2020-12_15 12:57:53\n",
      "EPOCH = 39 loss: 0.952, acc: 0.797, val_loss: 1.423, val_acc: 0.744\n",
      "\n",
      "================================================================================2020-12_15 12:58:00\n",
      "EPOCH = 40 loss: 0.908, acc: 0.803, val_loss: 1.343, val_acc: 0.765\n",
      "\n",
      "================================================================================2020-12_15 12:58:06\n",
      "finishing training...\n",
      "Training complete in 4m 4s\n",
      "    epoch      loss       acc  val_loss   val_acc\n",
      "0     1.0  7.785803  0.005774  7.445163  0.119145\n",
      "1     2.0  7.084351  0.191411  6.531834  0.291639\n",
      "2     3.0  6.306262  0.293269  5.850180  0.336391\n",
      "3     4.0  5.689270  0.359044  5.239167  0.427689\n",
      "4     5.0  5.135109  0.414836  4.753136  0.421576\n",
      "5     6.0  4.666792  0.440490  4.311813  0.469719\n",
      "6     7.0  4.234490  0.485365  3.898084  0.528650\n",
      "7     8.0  3.836374  0.523783  3.562246  0.543119\n",
      "8     9.0  3.514324  0.543715  3.306483  0.565698\n",
      "9    10.0  3.272735  0.556921  3.105244  0.574188\n",
      "10   11.0  3.072498  0.567817  2.941895  0.581470\n",
      "11   12.0  2.901383  0.576444  2.804728  0.590379\n",
      "12   13.0  2.767302  0.586991  2.682871  0.603717\n",
      "13   14.0  2.632904  0.600227  2.573406  0.614186\n",
      "14   15.0  2.517037  0.609733  2.481636  0.620475\n",
      "15   16.0  2.418065  0.619143  2.400453  0.628487\n",
      "16   17.0  2.324149  0.627621  2.318220  0.637150\n",
      "17   18.0  2.237058  0.636989  2.246829  0.643704\n",
      "18   19.0  2.158857  0.644323  2.178703  0.652174\n",
      "19   20.0  2.079949  0.653273  2.120313  0.658980\n",
      "20   21.0  2.008425  0.658950  2.093735  0.657707\n",
      "21   22.0  1.936997  0.667249  2.003060  0.669666\n",
      "22   23.0  1.862279  0.674603  1.948800  0.677244\n",
      "23   24.0  1.804200  0.680009  1.904132  0.683276\n",
      "24   25.0  1.743539  0.686685  1.863146  0.687342\n",
      "25   26.0  1.663603  0.697544  1.811239  0.691821\n",
      "26   27.0  1.609760  0.702511  1.767373  0.699901\n",
      "27   28.0  1.549700  0.709026  1.719256  0.708135\n",
      "28   29.0  1.485288  0.718541  1.700948  0.711433\n",
      "29   30.0  1.429099  0.725876  1.666769  0.714099\n",
      "30   31.0  1.372892  0.732802  1.611359  0.721917\n",
      "31   32.0  1.316053  0.742022  1.612513  0.719944\n",
      "32   33.0  1.262016  0.748856  1.543277  0.733273\n",
      "33   34.0  1.210503  0.755820  1.511122  0.735983\n",
      "34   35.0  1.150586  0.765573  1.483806  0.743476\n",
      "35   36.0  1.098552  0.772220  1.481392  0.740601\n",
      "36   37.0  1.048129  0.781438  1.435194  0.749569\n",
      "37   38.0  1.000101  0.787697  1.401873  0.755694\n",
      "38   39.0  0.951878  0.796895  1.423459  0.744392\n",
      "39   40.0  0.908292  0.802841  1.342724  0.764645\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/torchtext/data/batch.py:23: UserWarning: Batch class will be retired soon and moved to torchtext.legacy. Please see the most recent release notes for further information.\n",
      "  warnings.warn('{} class will be retired soon and moved to torchtext.legacy. Please see the most recent release notes for further information.'.format(self.__class__.__name__), UserWarning)\n"
     ]
    }
   ],
   "source": [
    "# 开始训练\n",
    "df_history = train_model(transformer, EPOCHS, train_dataloader, val_dataloader, print_trainstep_every)\n",
    "print(df_history)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 432x288 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEWCAYAAABsY4yMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAyQklEQVR4nO3deXxU5fX48c/JAklYBAHZQhJsFUUQ1Ki4Vqv2i0hFragYW1FKKi7FBRWNWmvFtl+tCy2CQUUrwQ2/VH9qrVXwW/XrFpTVHUkgohAQEAxbyPn98dyESTIzmQy5mSXn/XrNa2bunbn3zIWceebc5z6PqCrGGGOST0qsAzDGGOMPS/DGGJOkLMEbY0ySsgRvjDFJyhK8McYkKUvwxhiTpCzBm4iIyD9F5OKWfm0siUiZiJzqw3ZVRH7sPZ4hIrdG8too9lMgIq9GG2eY7Z4kIhUtvV3T+tJiHYDxj4hsDXiaBewAdnvPf6OqJZFuS1VP9+O1yU5VL2uJ7YhIHrASSFfVam/bJUDE/4am7bEEn8RUtWPtYxEpA36tqq81fJ2IpNUmDWNM8rASTRtU+xNcRG4UkW+BWSLSVUReFJFKEdnoPc4OeM8bIvJr7/FYEXlLRO7xXrtSRE6P8rX9ReQ/IrJFRF4TkWkiMjtE3JHE+AcRedvb3qsi0j1g/S9FpFxENohIUZjjc7SIfCsiqQHLzhaRJd7jo0TkHRHZJCLfiMjfRKRdiG09JiJ3Bjy/3nvPGhG5tMFrzxCRj0TkexFZLSK3B6z+j3e/SUS2isgxtcc24P3HisgHIrLZuz820mMTjogc7L1/k4gsF5EzA9aNEJGPvW1+LSKTvOXdvX+fTSLynYi8KSKWb1qZHfC2qxewL5ALFOL+L8zynucA24C/hXn/0cBnQHfgv4FHRESieO0c4H2gG3A78Msw+4wkxguBS4D9gHZAbcIZCEz3tt/H2182Qajqe8APwE8bbHeO93g3cI33eY4BTgEuDxM3XgzDvXhOAw4AGtb/fwB+BXQBzgAmiMhZ3roTvfsuqtpRVd9psO19gZeAqd5nuxd4SUS6NfgMjY5NEzGnA/8PeNV731VAiYgM8F7yCK7c1wkYBMz3ll8HVAA9gJ7AzYCNi9LKLMG3XTXA71R1h6puU9UNqvqcqlap6hZgCvCTMO8vV9WZqrobeBzojftDjvi1IpIDHAncpqo7VfUt4IVQO4wwxlmq+rmqbgOeAYZ6y88FXlTV/6jqDuBW7xiE8iQwBkBEOgEjvGWo6kJVfVdVq1W1DHgoSBzBnOfFt0xVf8B9oQV+vjdUdamq1qjqEm9/kWwX3BfCF6r6hBfXk8CnwM8DXhPq2IQzDOgI/Mn7N5oPvIh3bIBdwEAR6ayqG1X1w4DlvYFcVd2lqm+qDXzV6izBt12Vqrq99omIZInIQ14J43tcSaBLYJmigW9rH6hqlfewYzNf2wf4LmAZwOpQAUcY47cBj6sCYuoTuG0vwW4ItS9ca/0cEWkPnAN8qKrlXhwHeuWHb7047sK15ptSLwagvMHnO1pEFnglqM3AZRFut3bb5Q2WlQN9A56HOjZNxqyqgV+Ggdv9Be7Lr1xE/ldEjvGW3w18CbwqIl+JyOTIPoZpSZbg266GranrgAHA0aramT0lgVBll5bwDbCviGQFLOsX5vV7E+M3gdv29tkt1ItV9WNcIjud+uUZcKWeT4EDvDhujiYGXJkp0BzcL5h+qroPMCNgu021ftfgSleBcoCvI4irqe32a1A/r9uuqn6gqqNw5Zt/4H4ZoKpbVPU6Vd0fOBO4VkRO2ctYTDNZgje1OuFq2pu8eu7v/N6h1yIuBW4XkXZe6+/nYd6yNzHOBUaKyPHeCdE7aPr//xxgIu6L5NkGcXwPbBWRg4AJEcbwDDBWRAZ6XzAN4++E+0WzXUSOwn2x1KrElZT2D7Htl4EDReRCEUkTkfOBgbhyyt54D9fav0FE0kXkJNy/0VPev1mBiOyjqrtwx6QGQERGisiPvXMtm3HnLcKVxIwPLMGbWvcDmcB64F3glVbabwHuROUG4E7gaVx//WDuJ8oYVXU5cAUuaX8DbMSdBAyntgY+X1XXByyfhEu+W4CZXsyRxPBP7zPMx5Uv5jd4yeXAHSKyBbgNrzXsvbcKd87hba9nyrAG294AjMT9ytkA3ACMbBB3s6nqTlxCPx133B8EfqWqn3ov+SVQ5pWqLsP9e4I7ifwasBV4B3hQVRfsTSym+cTOe5h4IiJPA5+qqu+/IIxJdtaCNzElIkeKyI9EJMXrRjgKV8s1xuwlu5LVxFov4H9wJzwrgAmq+lFsQzImOViJxhhjkpSVaIwxJknFVYmme/fumpeXF+swjDEmYSxcuHC9qvYIts7XBC8i1wC/xl2ksRS4JPDqyYby8vIoLS31MyRjjEkqItLwCuY6vpVoRKQv8FsgX1UHAanABX7tzxhjTH1+1+DTgEwRScNNOLHG5/0ZY4zx+JbgVfVr4B5gFe7Kwc2q2mh6MREpFJFSESmtrKz0KxxjjGlzfKvBi0hX3EUr/YFNwLMicpGq1pvMQVWLgWKA/Px867NpTJLZtWsXFRUVbN8e8vSbiUBGRgbZ2dmkp6dH/B4/T7KeCqxU1UoAEfkf4Fgg6Gw9xpjkVFFRQadOncjLyyP0nDAmHFVlw4YNVFRU0L9//4jf52cNfhUwzBvDW3Cz3nzS0jspKYG8PEhJcfclNgWxMXFl+/btdOvWzZL7XhARunXr1uxfQb614FX1PRGZC3wIVAMf4ZViWkpJCRQWQpU3XUR5uXsOUFAQ+n3GmNZlyX3vRXMMfe1Fo6q/U9WDVHWQqv7SmyqtxRQV7Unutaqq3HJjjGnrEnqoglWrmrfcGGPakoRO8DkNJzxrYrkxJv619Hm1TZs28eCDDzb7fSNGjGDTpk3Nft/YsWOZO3dus9/nh4RO8FOmQFZW/WVZWW65MSbx1J5XKy8H1T3n1fYmyYdK8NXV1WHf9/LLL9OlS5fodxwHEjrBFxRAcTHkelMNp6W553aC1Zj4ddJJjW+1+femm4KfV5s40T1ev77xe5syefJkVqxYwdChQznyyCM54YQTOPPMMxk4cCAAZ511FkcccQSHHHIIxcV7+oHk5eWxfv16ysrKOPjggxk/fjyHHHIIP/vZz9i2bVtEn/X111/nsMMOY/DgwVx66aXs2LGjLqaBAwdy6KGHMmnSJACeffZZBg0axJAhQzjxxBPDbTZiCZ3gwSXzsjJ4/HGorobs7FhHZIyJVkWIWXI3bIh+m3/605/40Y9+xKJFi7j77rv58MMPeeCBB/j8888BePTRR1m4cCGlpaVMnTqVDUF29sUXX3DFFVewfPlyunTpwnPPPdfkfrdv387YsWN5+umnWbp0KdXV1UyfPp0NGzYwb948li9fzpIlS7jlllsAuOOOO/jXv/7F4sWLeeGFF6L/wAHiarjgvXHuufDpp9CMawCMMTHwxhuh1+XkuLJMQ7W/0rt3D//+SBx11FH1LhaaOnUq8+bNA2D16tV88cUXdOvWrd57+vfvz9ChQwE44ogjKCsra3I/n332Gf379+fAAw8E4OKLL2batGlceeWVZGRkMG7cOEaOHMnIkSMBOO644xg7diznnXce55xzzt59SE/Ct+BrZWXBXXfZCVZjEllrnFfr0KFD3eM33niD1157jXfeeYfFixdz2GGHBb2YqH379nWPU1NTm6zfh5OWlsb777/Pueeey4svvsjw4cMBmDFjBnfeeSerV6/miCOOCPpLorkSP8EHnHLX3DwW31DCv/8d66CMMdEIPK8m4u739rxap06d2LJlS9B1mzdvpmvXrmRlZfHpp5/y7rvvRr+jBgYMGEBZWRlffvklAE888QQ/+clP2Lp1K5s3b2bEiBHcd999LF68GIAVK1Zw9NFHc8cdd9CjRw9Wr1691zEkdommwaWssqqcA+8p5Nm5cNpXdqbVmERUUNCyHSW6devGcccdx6BBg8jMzKRnz55164YPH86MGTM4+OCDGTBgAMOGDWux/WZkZDBr1ixGjx5NdXU1Rx55JJdddhnfffcdo0aNYvv27agq9957LwDXX389X3zxBarKKaecwpAhQ/Y6hriadDs/P1+bNaNTXl7Qgl0ZuWxeVEYLHB9jzF765JNPOPjgg2MdRlIIdixFZKGq5gd7fWKXaEJcsprDKh55pJVjMcaYOJPYCT7EGdUNWTnMng02/LQxxi9XXHEFQ4cOrXebNWtWrMOqJ7Fr8FOm1B9OEiAri2+vmkLWbPjySxg0KHbhGWOS17Rp02IdQpMSuwVfe8q9tiXfsSMUF3PIXQWUl1tyN8a0bYmd4MEl+fJyGD0aOnWCMWNISYHUVNi1C7ZujXWAxhgTG4mf4GuNHAnffAMffQTADz+4Tjb33BPbsIwxJlaSJ8Gffrq7MuLFFwHo0MGVaGbNgt27YxybMcbEQPIk+B49YNiwugQPMG6c60nZp4/N2WpMwojxRMsdO3YMua6srIxBCXRyL3kSPLgyTWmpK9UAtSN6rlvXcmNLG2N85MeA8G1YYneTbOjnP3cTsr78Mowbx+9+1/gltXO22pjxxsTA1VfDokWh17/7LuxoMHVzVZX7OT5zZvD3DB0K998fcpOTJ0+mX79+XHHFFQDcfvvtpKWlsWDBAjZu3MiuXbu48847GTVqVHM+Cdu3b2fChAmUlpaSlpbGvffey8knn8zy5cu55JJL2LlzJzU1NTz33HP06dOH8847j4qKCnbv3s2tt97K+eef36z9RcO3FryIDBCRRQG370Xkar/2B7iie05OXZnG5mw1JsE0TO5NLY/A+eefzzPPPFP3/JlnnuHiiy9m3rx5fPjhhyxYsIDrrruO5g7bMm3aNESEpUuX8uSTT3LxxRezfft2ZsyYwcSJE1m0aBGlpaVkZ2fzyiuv0KdPHxYvXsyyZcvqRpD0m28teFX9DBgKICKpwNfAPL/2h9uRK9M89hhs305OTkbQsaVtSGFjYiRMSxsIOb4UublRDwR/2GGHsW7dOtasWUNlZSVdu3alV69eXHPNNfznP/8hJSWFr7/+mrVr19KrV6+It/vWW29x1VVXAXDQQQeRm5vL559/zjHHHMOUKVOoqKjgnHPO4YADDmDw4MFcd9113HjjjYwcOZITTjghqs/SXK1Vgz8FWKGqQf7lWtjIke4n3Rtv2JytxiQan/5oR48ezdy5c3n66ac5//zzKSkpobKykoULF7Jo0SJ69uwZdBz4aFx44YW88MILZGZmMmLECObPn8+BBx7Ihx9+yODBg7nlllu44447WmRfTWmtBH8B8GSwFSJSKCKlIlJaWVm593s6+WT3H+LFF+sudO3b163q0sXmbDUmrvkxIDyuTPPUU08xd+5cRo8ezebNm9lvv/1IT09nwYIFlAf71dCEE044gRLv5O/nn3/OqlWrGDBgAF999RX7778/v/3tbxk1ahRLlixhzZo1ZGVlcdFFF3H99dfz4Ycf7tXniZiq+noD2gHrgZ5NvfaII47QFnHmmaq5uao1NXWLjj1WdfDgltm8MSZyH3/8caxDUFXVQYMG6UknnaSqqpWVlTps2DAdNGiQjh07Vg866CBduXKlqqp26NAh5DZWrlyphxxyiKqqbtu2TceOHauDBg3SoUOH6vz581VV9Y9//KMOHDhQhwwZov/1X/+lGzZs0FdeeUUHDx6sQ4YM0fz8fP3ggw+i+gzBjiVQqiFyqu/jwYvIKOAKVf1ZU69t9njwocyc6bpWLV1aNyDN3/4GV11Vb5ExphXYePAtJx7Hgx9DiPKMb0aMcPcBFz2NHu2um3iydSMxxpiY8bUfvIh0AE4DfuPnfhrp2xcOP9wl+MmTAejZE84+G2pqWjUSY0wCWrp0Kb/85S/rLWvfvj3vvfdejCKKjq8JXlV/ALr5uY+Qfv5z+MMfYP166N4dgLlzYxKJMW2eqiIisQ4jYoMHD2ZRuAuyYiCacnpyDVUQaORI11z/5z8brVq3LgbxGNNGZWRksGHDhqgSlHFUlQ0bNpCRkdGs9yXXUAWBDj8cevVyZZqAn1oTJ8Izz0BFhRsz3hjjr+zsbCoqKmiRbtBtWEZGBtnZ2c16T/Im+JQUOOMMePZZN/NHejoAJ5wAU6e6i+JOOSW2IRrTFqSnp9O/f/9Yh9EmJW+JBlyZ5vvv4a236hadcYab+GnOnBjGZYwxrSC5E/ypp7o6zKhRdWNLZ/5PCWefDc89t1fjFxljTNxL7gT//PNuTOktW+qNLX1tzxI2b4ZXXol1gMYY45/kTvBFRY07vldVcejTRcyZYzV4Y0xyS96TrBBy4HdZvYoxY1o5FmOMaWXJ3YIPNfB7Tg47dsDdd1uZxhiTvJI7wYcZWzo9Hf76V3czxphklNwJvnZs6dqWfGZm3djSKSkwZgy8+qobzcAYY5JNcid4cEm+vBxuvBF27nQTgnj22Qeqq6FHDzdTmE3cboxJJsmf4Gv9+tewezfMmgW4ZB44C5jXg9KSvDEmabSdBP/jH7t+kTNnQk0NRUVu6tZAVVWuZ6UxxiSDtpPgwTXRy8vh3/8O1YMy5HJjjEk0bSvBn3WWK7gXF4frQWmMMUmhbSX4du1g7Fh44QX+MumbRj0o09Pr1+WNMSaRta0ED+5ka3U1v9jyGMXFkJsLIi73d+2KXeFqjEkabS/BH3ig6yo5cyYFY2ooK3PD1fz9726mp5deinWAxhjTMnxN8CLSRUTmisinIvKJiBzj5/4iVlgIK1fC66/XLfrFL6BfP7jvvhjGZYwxLcjvFvwDwCuqehAwBPjE5/1F5uyzoVs3d1WrJy0NrrwS3nwTVq+OYWzGGNNCfEvwIrIPcCLwCICq7lTVTX7tr1nat3cnW//xD1i7tm7xZZdBWZlryRtjTKLzswXfH6gEZonIRyLysIh08HF/zTN+vBun4LHH6hZ17gx9+7rHDYeRN8aYRONngk8DDgemq+phwA/A5IYvEpFCESkVkdJWnXV9wAA48cS6K1tr7doFp50Gd9zReqEYY4wf/EzwFUCFqr7nPZ+LS/j1qGqxquaran6PHj18DCeIQw6BFStcAd4bbSw9HTIy4MEHYfv21g3HGGNakm8JXlW/BVaLyABv0SnAx37tr9lKSuDxx93jgPlaKSnhmmugshLmzIltiMYYszdEVf3buMhQ4GGgHfAVcImqbgz1+vz8fC0tLfUtnnry8lxSbyg3F11ZxtChLu8vXuwuhDLGmHgkIgtVNT/YOl+7SarqIq/8cqiqnhUuube6MKONicDVV8PSpfW6yhtjTEJJ7km3w8nJCd6C90YbGzMGvv8e8oN+LxpjTPxre0MV1Ao2X2u7dnWjjWVkQPfuMHQopKTYjE/GmMTTdlvwBQXuvqjIlWvS0lxGv/BCwCXzwsI9k4LUnoMNfKsxxsSzttuCB5epa0cbe/hhWLPGzcINNuOTMSbhte0EH+iCC6B3b7j3XiDsOVhjjEkIluBrtWvnRht79VVYtsxmfDLGJDxL8IF+8xvIzIT77w96DtZmfDLGJBJL8IG6dYOLL4bZsyk4dW29GZ86d4YJE+wEqzEmcViCb+jqq2HHDpg+vd452M2b4YEHYh2cMcZEzhJ8QwMGwMiRQUcb27IF7rnHXQBljDHxzhJ8MLWjjTW4sumzz+D6613uN8aYeGcJPpiTT4YhQ1yXyYDB2PLzYfhw+Mtf4IcfYhifMcZEwBJ8MCJw7bXw8cd1Fz7Vuu02WL8eZsyIUWzGGBMhS/ChXHAB7LMPnHVWvcFojjkGTjkF7r4btm2LdZDGGBOaJfhQnn3WjU2wfXujCUFuvdVNBtWaMwwaY0xz+TrhR3O16oQfTQkzIQhlZa0djTHGBBWzCT8SWgSD0fz1r9Crlw0nbIyJT5bgQ2liMJqSEndN1Nq1jSo4xhgTFyzBhxJsMJr27esGoykqcle4BrLhhI0x8cQSfCgFBdQbjCYlBbKz6yYEseGEjTHxztcELyJlIrJURBaJSJycPW2GwMFoiothxQrXu4YmKzjGGBNzrdGCP1lVh4Y6y5swxo6FQw+FyZNhx46gFZysLBtO2BgTP6xEE6nUVDfS2MqV8Ne/Nqrg5Oa65zacsDEmXvjaD15EVgIbAQUeUtXiIK8pBAoBcnJyjigP1vc8nowYAf/3f/Dll26S7gAffOCSfX5i/1YxxiSQWPaDP15VDwdOB64QkRMbvkBVi1U1X1Xze/To4XM4LeDuu924wXfcUW9xdTWMHg3jx8Pu3TGKzRhjAvia4FX1a+9+HTAPOMrP/bWKQw5xWXz6dPj887rFaWnw5z/DokXw6KOxC88YY2r5luBFpIOIdKp9DPwMWObX/lrV738PGRlwww31Fp93HpxwAtx8M2zaFJvQjDGmlp8t+J7AWyKyGHgfeElVX/Fxf62nZ0+46SZ4/vl6YxXInBIeeAA2bGhUwTHGmFaX5teGVfUrYIhf24+53r3dGdW1a91zb6yCw4ph4sQCevWKbXjGGGOjSUbLRps0xsQBG03SD02MVaAKEye6ao6NNmmMiQVL8NFqYqyCJ55wwwmvW2ejTRpjYsMSfLSCjVWQkVE3VsFtt9Wbrxuw0SaNMa3LEny0go022aePm8sVG23SGBN7luD3RuBok3//O3z1FcycCdhok8aY2LME31IuvBB++lPXP37tWhtt0hgTcxEleBGZKCKdxXlERD4UkZ/5HVxCEYEHH3SF9kmTbLRJY0zMRdqCv1RVv8cNN9AV+CXwJ9+iSlQDBrjhC2bPhgUL6lVwysqgshKuvTbWQRpj2opIE7x49yOAJ1R1ecAyE+jmm2H//WHCBNixo96q1avhvvvg7bdjFJsxpk2JNMEvFJFXcQn+X94gYjVNvKdtysyEadPgs8/cBCEBfv97d5K1sBB27oxRfMaYNiPSBD8OmAwcqapVQDpwiW9RJbrhw+Hcc+H226Fv37pLWTs+X8K0afDxx41yvzHGtLhIE/wxwGequklELgJuATb7F1YS+MlP3Cwga9bUu5R15OYSzj0X7rzTjTppjDF+iTTBTweqRGQIcB2wAvi7b1Elg2BNdO9S1gcegNdeg1decWPU2Fg1xhg/RJrgq9UNOzkK+JuqTgM6+RdWEghzKWufPm7u7sJC17C3sWqMMX6INMFvEZGbcN0jXxKRFFwd3oTSxKWsRUWuQR/IxqoxxrSkSBP8+cAOXH/4b4Fs4G7fokoGwS5lbd++7lJWG6vGGOO3iBK8l9RLgH1EZCSwXVWtBh9Ow0tZ09NdF8qRIwEbq8YY479Ihyo4Dzev6mjgPOA9ETnXz8CSQuClrG+/Dd9/D9dfDwRv4Gdm2lg1xpiWE2mJpgjXB/5iVf0VcBRwq39hJaEjj4RJk9xok6+9FnS04QkTbKwaY0zLiWhOVhFZqqqDA56nAIsDl4V5bypQCnytqiPDvTah5mSNxrZtMHSou4x16VLo2LFu1dat9Z4aY0xEWmJO1ldE5F8iMlZExgIvAS9H+N6JwCcRvja5ZWbCo4+6PpE331xvVceOrrvkU08Fn8vbGGOaK9KTrNcDxcCh3q1YVW9s6n0ikg2cATy8N0EmleOOg6uuchO2vvlmvVWVlTB+PFx6qSvbG2PM3oh4wg9VfU5Vr/Vu8yJ82/3ADYQZmExECkWkVERKKysrIw0nsd11F/TvD6NHu24z3qWs+/27hPvug/nz3XhlxhizN8ImeBHZIiLfB7ltEZHvm3jvSGCdqi4M9zpVLVbVfFXN79GjRxQfIQF16ODmbl271o0hHHAp67jMEk4/HW68ET7/PNaBGmMSWdgEr6qdVLVzkFsnVe3cxLaPA84UkTLgKeCnIjK7heJOfHPmNF5WVYUUFfHww65nzeDBNk6NMSZ6vs3Jqqo3qWq2quYBFwDzVfUiv/aXcMJcyrpgAeze7Trb2Dg1xpho2aTbsRLmUtaiokaTQdk4NcaYZmuVBK+qbzTVB77NCXYpa2oqTJli49QYY1qEteBjpeGlrPvs4+oy330XsnGfnd26IRpjEpsl+FgKHKvmu+/gzDPhmmt4uGBBo8Y9QLdu7jvAGGMiYQk+XqSkwBNPwIEHcupDoymZUlbXuM/NhUsugUWL6sYqM8aYJqXFOgAToHNn+Mc/4KijOOuxszhr+duuz7xnn33cTdUlfmOMCccSfLw58EA3IM3pp8N++7kBynJyYMoU7r23oC6xV1dDmv3rGWPCsBJNPNqwwU0QUlVVryO8zHEd4d97D/r2dTe7EMoYE4ol+HhUVAS7dtVfFtAR/p13YN06WLPGLoQyxoRmCT4eNdER/v77G6+yC6GMMQ1Zgo9HoTrCd+4MqnYhlDEmIpbg41Goq1w3b4brriOnX/BZuGzCbmNMIEvw8ajhVa65ufDYY/Db38J99/Ha/oV0zKx/xVNWlpvy9d13YxOyMSb+WIKPV4FXuZaVwUUXueL7Lbfw4zce5qs+x1ORmstuUlidmse/Li7hf/8XTj7ZdaU3xhhL8IlEBP7wB7jgAnqseJe+u1eRgpK9u5zjHy/kkVNKOPRQ+MUv3JWveXnWjdKYtswSfCJ6553Gy6qq6PynIubPh0MPdRWd8nLrRmlMW2YJPhGF6UbToYMbt6wh60ZpTNtjCT4Rheou06kTVFezenXw1daN0pi2xRJ8IgrVjfL77+Gkkzi6T/AM36WLK9kYY9oGS/CJKFg3yscfd0X2xYt5Y/NQ7ku9jpXksZsUVpLHhZSwcaM7AbtxY6w/gDGmNYjGUZMuPz9fS0tLYx1GYvviCzjtNHdmNUB1uyz+dW4xZz9bQO/e7qTrzJmubOMNVklBQYxiNsZETUQWqmp+sHXWgk82BxwQtA6TtrOKM94u4u23XaN/yhTrZWNMsvMtwYtIhoi8LyKLRWS5iPzer32ZBsKcZT3ySJfUt22rv8p62RiTfPxswe8AfqqqQ4ChwHARGebj/kytcIPSTJvG16t2M4aSejX6MZRYLxtjkoxvCV6drd7TdO8WPwX/ZBasl01GBhx8MFx5JStTf8QjjCOPclJQ8ihnJoUUUMKbb8YmZGNMy/O1Bi8iqSKyCFgH/FtV3wvymkIRKRWR0srKSj/DaTuC9bJ5+GFYtgyefpq+WkEmO+q9pQNV3CVFXHQR7Nzp6vE21IExia1VetGISBdgHnCVqi4L9TrrRdNKUlKCnohVEZYtrmHJEnfStapqz7qsLPedYT1tjIkvMe9Fo6qbgAXA8NbYn2lCiBq9dOrE4D4bKCqCUVX1a/SjqkrsJKwxCcbPXjQ9vJY7IpIJnAZ86tf+TDOEuxI2L4/7ys/iYcY3qtEfV16CqpVvjEkUfrbgewMLRGQJ8AGuBv+ij/szkQp1JeyyZXD66ZzN82RRvx9lB6q4iyLmzHHlG+tDb0z8sytZTSMqKUiQDk+K0D+3puFFsoD7jigr8z82Y0x9Ma/Bm8QiuSFq9JkZ9Ct/y/rQG5MgLMGbxoLV6NPSICWFNzmBJ/hV4z70UsL27e6lVqM3Jj5YgjeNhZr0e+1adnToSio19V7egSr+0r6IjAyXzMeNsxq9MfHAErwJruGk3wUF0KED7as2BX35ftvK4fXXuWFSDefssC6WxsSDtFgHYBJMTk6joYgBV4859VRK2Y9ubKQduwDqSjiF5QB2lZQxrcla8KZ5gtXns7LgkUdg9mz2ZXNdcq/VgSr+nOqa8Bs3Wo3emNZiCd40T7D6fHExjB0LBQW0Y2fQt/XdXQ4vvsixR+zg5YtKeKM8j2pN4Y3yPF67pMSSvDE+sH7wpmXl5YUu4dTUUEUm7dhJGrvrVv1AFjd1K2bqeivhGNNc1g/etJ5QJZxHH4WXX6aGlHrJHVwJ5/oNk9m0Cb78Et66vISKtDxqJIWKtDzeutya98ZEw06ympZVO9xkUVHQCV+zqAr6tmwqWJN/Iu+v6MPZPE8mrlN99u5yuk4v5C3g+AethW9Mc1gL3rS8YF0sPVXdgl8luytzH3pkbOFCnq5L7rU6UEVe8Z5+ltbCNyYyluBNq+r4wBSq29Uv4VS3y6LdzGm0W/YRNUjQ9/XdXc4LY57kqVOKOWx6Idm73ZW02bvLOWx6oSV5Y4KwBG9aV0EBaY/W74WT9uiemUTWpAZv4deQwplPXcj5839DhwZlnnotfOuDaUwdS/Cm9YUp4ZQVTuEH6rfwfyCLdy57nOo33wm5yb67y9l+91SqL60/lnH1pTZOgmm7LMGbuHL8gwV8NKGYitRcahAqUnP5aEIxx0+/iLTjh/F1am7Q9wmQccNE0nbWb92n7axi68SAcRKshW/aEEvwJu4c/2AB2dVlpGgN2dVl9XrPhGrhLxj+xyAj2DsdNpTDvHnw0EPWwjdtiiV4k1BCtfBP/udkygneulcEzjkHLrvMWvimTbEEbxJOqBb+vd2Ct+6v23cWd496K3wL/+674dZbqb50fPgWvn0BmARiFzqZpHH0AwVceQn8blcROaxiFTn8Pn0Kp04tcOd1JZc8Gg+jUE0a6TfcADT+g0jbWcX2CVeT0a8ffPAB3HorbPPmq60d7B7qnSg2Jl741oIXkX4iskBEPhaR5SIy0a99GQMux546q4CTcstIkxpOyi3j1FkFdbk3VAv/6i6PkZu5LmQf/Iwt6+EnP4FJk/Yk91pVVdQb7N5a+CaO+FmiqQauU9WBwDDgChEZ6OP+jAnXA9O18NOLKcPV78vI5cr0Yo79WwGffdeDVQTvg7+G3rw66dWQJR4tL4cZM2D6dNeit+msTLxQ1Va5Ac8Dp4V7zRFHHKHG+Gn2bNXcXFURdz979p51V3WbrVvJUnXpWRV0K1k6LnO2gupKcuutq73tJC3o8rpbbm7TOzcmSkCphsiprXKSVUTygMOA91pjf8aEEk0L/6TiAhYvhpsJXuIZy2O8+9Di8C38yy+HX/86fOveyjumpYXK/C11AzoCC4FzQqwvBEqB0pycHJ+/64wJL1wjOzdXdQyzdSW5uhvRleTqGGZrerqGbeHvDte6328/1W+/dTvKqv/rQbOy6gdgvwBMEIRpwfud3NOBfwHXRvJ6K9GYeBYqBz/8sOpLL7nkH6zEcxGPaw0SvoyTkhJ8eW2jJ5IvANMmhUvwfvaiEeAR4BNVvdev/RjTWkLNVjhuHIwYAf+XW8B46pd4xlPMbH5FeYgTuJUp+8EDD7iaUTCrVsGgQTB+vOuxE8h68JimhMr8e3sDjgcUWAIs8m4jwr3HWvAmkYVqZP/xj6Fb9xcyW6dNU924T27QFvyOzM6qo0aFb/3fc4/qTTepZmZaiacNIlYlmubeLMGbRBcqh4aq3++7r/srDPUFcFW3gA0ES+6pqeGTf69eqps3W4kniVmCNybGwuXXsjL3hRDsCwBUR49W/duxwb8A3pwwW7Wy0m0gXKIP9UVQ24WzNshQLXxr/cctS/DGxIGmeugEy79duuxZF+wLoC4/h9pAjx6uRhQu+Y8apXrWWart2gX/BrIePnHNErwxca6pHBqugX7FFaolZ4Rp4auG/gLIylI9+ODQG09N1bp+oA1vPXuqrlmj+sQTVv6JIUvwxiSAaFr4mZl7cmvYFv7s2bqrXf0kvKtdhN8g0d4Cr2tpqoVvvwCiZgnemAQXroW/bVv4/DxmjLv9KrX+F8DY9Nl78miob5Dc3PDln2nTwif5E05QHT68cfknM1O1uFj1u+9UH3qo6R5AJiRL8MYkgWhb+P36hc6/vXurVlervjkhTImnqfpRqJ136qQ6bFj4L4Bwtz59VGtqmv7wbZwleGOS3N7U8Dt1Us3ICF/ieXPCbF2d6tatTs3dU9vf253fd1/4JN+vn+rxx4c+AVy7/70p/yT4l4cleGPagGha+N26qV5+eej8KqI6dWoE51Cj2Xntt0eo9fvuq3ruuaGHcejQQfXss5tO/uGCT4LrAyzBG9PGRVtlCXeLtAv9XifZaE4Ap6WpnnRS4+3W3rp3V33qKXe/Vx8uggPv868DS/DGmCavYwqWY//wh/B59KqrVC+7zJV4om7hN7U+3C+AcMn/+OOb/8UQeJs0SbWwsOkPFyr2Vvp1YAneGNOkcMMsBMt/7duHbiCDat++LXSONFyijLb806eP6vLl7ixzsPXp6e4DhvpwnTurzpihev31jb8A2rdXvfJK1a5dw8e21wfGsQRvjIlauPxaXR2+Ed2jh+qQIY2vlWr2hbDRtpL3Zv2uXf5cHwCq997rBolrgRa+JXhjzF6JpoLStavq2LGhL4Tt0kX1jTfcePp7lef87EUT6sPl5KiuXh36C0BENTs7+LpQByRYCz8CluCNMb7x4xypD5UMfz5cuBJRuPd+9VXoDy7SrBAtwRtjfBVNIzg7282EFS7Jjx2r+qtfNS6HN6cbvK8fLpISULTdRyNkCd4YEzPRNoIzMtyUtaGS/377ufOcMe/GHu03TAv1srEEb4yJqWgbwTU10ZV44qK8EwnrRWOMSXbRVDLCte5ryzuXXBJ9N/ZEES7Bi1sfH/Lz87W0tDTWYRhj4khJCRQW1p9zPCvLTXheVATl5Y3fk5kJHTrA+vXBt9m7t5vP/OmnQ2+7oKBlP4dfRGShquYHW5fS2sEYY0xzFBS4hJubCyLuvjYBT5niEnKgrCyYORPWrXOvD+abb6BTJ7j00vrJHdzzoqI9z0tKIC8PUlLcfUlJS346f/mW4EXkURFZJyLL/NqHMaZtKCiAsjKoqXH3ta3rcMlfBHJygm+vWze4/HLYuTP4+vJy+NOf4IYbYPx491zV3RcW1k/y8fwF4FuJRkROBLYCf1fVQZG8x0o0xpiWFK68U1DgEnKwEk9qKuzeHXq7vXrBihUwb17sSzwxKdGo6n+A7/zavjHGNCVcCx9Cl3gefxw2bgxd4vn2W+jYEcaODV/iiXXrPuY1eBEpFJFSESmtrKyMdTjGmCQTqrxTuy7UF0CXLqFLPN27w223QXV18PXl5XDyyXDJJbEt7/jai0ZE8oAXrURjjElE0ZZ42reHXbvcl0pDXbrA7be7E71Tp8K2bcG3HSnrRWOMMVGItsTzyCOu1R7Mpk1w9dXw5z/XT+7QuAfP3rIEb4wxYURb4glV3snJcTX8UPX9VataLnY/u0k+CbwDDBCRChEZ59e+jDEmVkJ9AYRq3d91F/TsGf4LoKX42YtmjKr2VtV0Vc1W1Uf82pcxxsSbaMs7U6a0XAxpLbcpY4wxgQoKQp8wrV1eVOTKMjk5Lrm3ZP95S/DGGBMj4b4AWoKdZDXGmCRlCd4YY5KUJXhjjElSluCNMSZJWYI3xpgkFVczOolIJRBkZAcAugMh5meJOYstOhZbdCy26CRrbLmq2iPYirhK8OGISGmoAXVizWKLjsUWHYstOm0xNivRGGNMkrIEb4wxSSqREnxxrAMIw2KLjsUWHYstOm0utoSpwRtjjGmeRGrBG2OMaQZL8MYYk6TiPsGLyHAR+UxEvhSRybGOpyERKRORpSKySERiOqGsiDwqIutEZFnAsn1F5N8i8oV33zWOYrtdRL72jt0iERkRo9j6icgCEflYRJaLyERveUyPXZi44uW4ZYjI+yKy2Ivv997y/iLynvc3+7SItIuj2B4TkZUBx25oa8fmxZEqIh+JyIvec3+OmarG7Q1IBVYA+wPtgMXAwFjH1SDGMqB7rOPwYjkROBxYFrDsv4HJ3uPJwJ/jKLbbgUlxcNx6A4d7jzsBnwMDY33swsQVL8dNgI7e43TgPWAY8Axwgbd8BjAhjmJ7DDg3Do7dtcAc4EXvuS/HLN5b8EcBX6rqV6q6E3gKGBXjmOKWqv4H+K7B4lHA497jx4GzWjOmWiFiiwuq+o2qfug93gJ8AvQlxscuTFxxQZ2t3tN076bAT4G53vKY/J8LE1vMiUg2cAbwsPdc8OmYxXuC7wusDnheQRz9B/co8KqILBSRwlgHE0RPVf3Ge/wt0DOWwQRxpYgs8Uo4MSkfBRKRPOAwXIsvbo5dg7ggTo6bV2pYBKwD/o37xb1JVau9l8Tsb7ZhbKpae+ymeMfuPhFpH4PQ7gduAGq8593w6ZjFe4JPBMer6uHA6cAVInJirAMKRd3vv7hoxXimAz8ChgLfAH+JZTAi0hF4DrhaVb8PXBfLYxckrrg5bqq6W1WHAtm4X9wHxSqWhhrGJiKDgJtwMR4J7Avc2JoxichIYJ2qLmyN/cV7gv8a6BfwPNtbFjdU9Wvvfh0wD/efPJ6sFZHeAN79uhjHU0dV13p/hDXATGJ47EQkHZdES1T1f7zFMT92weKKp+NWS1U3AQuAY4AuIlI7HWjM/2YDYhvulb1UVXcAs2j9Y3cccKaIlOFKzj8FHsCnYxbvCf4D4ADvDHM74ALghRjHVEdEOohIp9rHwM+AZeHf1epeAC72Hl8MPB/DWOqpTZ6es4nRsfNqoI8An6jqvQGrYnrsQsUVR8eth4h08R5nAqfhzhMsAM71XhaT/3MhYvs04AtbcHXuVj12qnqTqmarah4un81X1QL8OmaxPpscwdnmEbjeAyuAoljH0yC2/XE9exYDy2MdH/Ak7if7Llwdbxyuvvc68AXwGrBvHMX2BLAUWIJLpr1jFNvxuPLLEmCRdxsR62MXJq54OW6HAh95cSwDbvOW7w+8D3wJPAu0j6PY5nvHbhkwG6+nTYyO30ns6UXjyzGzoQqMMSZJxXuJxhhjTJQswRtjTJKyBG+MMUnKErwxxiQpS/DGGJOkLMGbpCciuwNGD1wkLTgqqYjkBY6QaUw8SWv6JcYkvG3qLlk3pk2xFrxps8SN5f/f4sbzf19EfuwtzxOR+d6AVK+LSI63vKeIzPPGGF8sIsd6m0oVkZneuOOveldOIiK/9cZyXyIiT8XoY5o2zBK8aQsyG5Rozg9Yt1lVBwN/w43yB/BX4HFVPRQoAaZ6y6cC/6uqQ3Bj2y/3lh8ATFPVQ4BNwC+85ZOBw7ztXObPRzMmNLuS1SQ9Edmqqh2DLC8DfqqqX3mDen2rqt1EZD3u8v9d3vJvVLW7iFQC2eoGqqrdRh5uKNoDvOc3AumqeqeIvAJsBf4B/EP3jE9uTKuwFrxp6zTE4+bYEfB4N3vObZ0BTMO19j8IGC3QmFZhCd60decH3L/jPf4/3Eh/AAXAm97j14EJUDeZxD6hNioiKUA/VV2AG3N8H6DRrwhj/GQtCtMWZHoz+9R6RVVru0p2FZEluFb4GG/ZVcAsEbkeqAQu8ZZPBIpFZByupT4BN0JmMKnAbO9LQICp6sYlN6bVWA3etFleDT5fVdfHOhZj/GAlGmOMSVLWgjfGmCRlLXhjjElSluCNMSZJWYI3xpgkZQneGGOSlCV4Y4xJUv8fAtFa2kBdlmEAAAAASUVORK5CYII=\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 432x288 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1KklEQVR4nO3deXhU9bnA8e9LWAPIEiIuIQlVXABZFEHbXvVaF7TWtdQlWqnW3KK4Ya1oetVqY2/1ioriEq8KShStvQqt1K1ul6otWAHBpSIECIIEMOxLQt77x+8EJjPnTALJyZlk3s/zzDMzZ5s3RznvOb9VVBVjjDHpq03UARhjjImWJQJjjElzlgiMMSbNWSIwxpg0Z4nAGGPSnCUCY4xJc5YITJMTkb+IyKVNvW2URKRMRE4K4bgqIgd7nx8Vkf9syLZ78TsFIvL63sZpWjexfgQGQEQ2xXzNBLYDO73v/6Gqpc0fVeoQkTLg56r6ZhMfV4F+qrqoqbYVkXxgCdBOVaubJFDTqrWNOgCTGlS1S+3nZBc9EWlrFxdjWhcrGjJJicgJIlIuIjeJyCrgKRHpISJ/FpEKEfnW+5wTs887IvJz7/NoEZklIv/tbbtERE7by237ish7IrJRRN4UkUkiMjUg7obEeKeI/M073usi0itm/SUislRE1opIUZLzM0JEVolIRsyyc0Rkvvd5uIh8ICKVIrJSRB4SkfYBx5osIr+N+X6jt8/XInJZ3LY/FJGPRWSDiCwXkdtjVr/nvVeKyCYRObb23Mbs/10RmS0i67337zb03Ozhee4pIk95f8O3IvJyzLqzRGSu9zd8JSIjg86zCZclAtMQ+wE9gTygEPf/zVPe91xgK/BQkv1HAF8AvYC7gSdERPZi22eBfwBZwO3AJUl+syExXgT8DNgXaA/8EkBE+gOPeMc/wPu9HHyo6t+BzcCJccd91vu8E7je+3uOBX4AXJkkbrwYRnrxnAz0A+LrJzYDPwW6Az8ExojI2d6647z37qraRVU/iDt2T+AVYKL3t00AXhGRrLi/IeHc+KjvPD+DK2oc4B3rPi+G4cDTwI3e33AcUBbwGyZsqmove9V54f5BnuR9PgHYAXRMsv0Q4NuY7+/gipYARgOLYtZlAgrstyfb4i4y1UBmzPqpwNQG/k1+Mf465vuVwKve51uBaTHrOnvn4KSAY/8WeNL73BV3kc4L2PY64KWY7woc7H2eDPzW+/wk8F8x2x0Su63Pce8H7vM+53vbto1ZPxqY5X2+BPhH3P4fAKPrOzd7cp6B/YEaoIfPdo/Vxmuv6F/2RGAaokJVt9V+EZFMEXnMKzrZgCuK6B5bPBJnVe0HVd3ifeyyh9seAKyLWQawPCjgBsa4KubzlpiYDog9tqpuBtYG/Rbu7v9cEekAnAv8U1WXenEc4hWXrPLiuAv3dFCfOjEAS+P+vhEi8rZXJLMe+EUDj1t77KVxy5YCB8Z8Dzo3ddRznvvg/pt967NrH+CrBsZrQmaJwDREfNOyG4BDgRGqug+7iyKCinuawkqgp4hkxizrk2T7xsS4MvbY3m9mBW2sqp/iLqSnUbdYCFwR0+e41j77ALfsTQy4J6JYzwIzgD6q2g14NOa49TUF/BpXlBMrF1jRgLjiJTvPy3H/zbr77LccOGgvfs+EwBKB2RtdcWXBlV55821h/6B3hz0HuF1E2ovIscCPQorxReAMEfm+V7F7B/X/W3kWuBZ3IfxDXBwbgE0ichgwpoExvACMFpH+XiKKj78r7m57m1feflHMugpckcx3Ao49EzhERC4SkbYicj7QH/hzA2OLj8P3PKvqSuAvwMNepXI7EalNFE8APxORH4hIGxE50Ds/JgKWCMzeuB/oBKwBPgRebabfLcBVuK7Flcs/j+vv4Od+9jJGVV0IXIW7uK8EvgXK69ntOeB44C1VXROz/Je4i/RG4HEv5obE8Bfvb3gLWOS9x7oSuENENuLqNF6I2XcLUAz8zWutdEzcsdcCZ+Du5tcCvwLOiIu7oe4n+Xm+BKjCPRWtxtWRoKr/wFVG3wesB94l8SnFNBPrUGZaLBF5HvhcVUN/IjGmNbMnAtNiiMjRInKQV5QwEjgLeDnisIxp8axnsWlJ9gP+F1dxWw6MUdWPow3JmJbPioaMMSbNWdGQMcakuRZXNNSrVy/Nz8+POgxjjGlRPvroozWqmu23rsUlgvz8fObMmRN1GMYY06KISHxv8l2saMgYY9KcJQJjjElzlgiMMSbNtbg6Aj9VVVWUl5ezbdu2+jc2vjp27EhOTg7t2rWLOhRjTDNrFYmgvLycrl27kp+fT/B8JyaIqrJ27VrKy8vp27dv1OEYY5pZqyga2rZtG1lZWZYE9pKIkJWVZU9UxqSo0lLIz4c2bdx7aWnTHj/URCAiI0XkCxFZJCLjfdbnepNrfCwi80Xk9Eb8VuOCTXN2/oxJTaWlUFgIS5eCqnsvLGzaZBBaIvBmKJqEm6yjP3ChNxdsrF8DL6jqUOAC4OGw4jHGmFQVdMevCjfdBFu21N1+yxYoKmq63w/ziWA4bv7Zxaq6A5iGGy0ylgL7eJ+74WZOMsaYtOF3xz96NAwYANnZsCJg3rhly5ouhjATwYHUnXO1nLpzogLcDlwsIuW4WZOu9juQiBSKyBwRmVNRUdHowJq6vK2yspKHH97zh5nTTz+dysrKxv24MSblBV1zVq2Ca69NvOOvroZFi+Dcc6FHD/9j5sZPXtoIUVcWXwhMVtUc4HTgGRFJiElVS1R1mKoOy872HSqjwcIobwtKBNXV1Un3mzlzJt27d9/7HzbGpDy/a85ll0HfvrD//rB2rf9+VVVQUgIPPgiZmXXXZWZCcXHTxRhmIlhB3cm3c0icHPtyvCn2VPUDoCPQq7E/fMIJia/a6/TNN/uXt117rfu8Zk3ivvUZP348X331FUOGDOHoo4/m3/7t3zjzzDPp399ViZx99tkcddRRDBgwgJKSkl375efns2bNGsrKyjj88MO54oorGDBgAKeccgpbt24N/L3HH3+co48+msGDB3PeeeexxfuDvvnmG8455xwGDx7M4MGDef/99wF4+umnGTRoEIMHD+aSSy6p/w8yxuyRoDv+b7+FceMSrzk7dsDXX8Nvf+uSgZ/aO/6CApcQ8vJAxL2XlLjlTUZVQ3nh+igsBvoC7YF5wIC4bf4CjPY+H46rI5Bkxz3qqKM03qefflrn+/HHJ74mTXLrRFRdXk58qapWVCTuW58lS5bogAEDVFX17bff1szMTF28ePGu9WvXrlVV1S1btuiAAQN0zZo1qqqal5enFRUVumTJEs3IyNCPP/5YVVVHjRqlzzzzTODv1e6vqlpUVKQTJ05UVdWf/OQnet9996mqanV1tVZWVuqCBQu0X79+WlFRUScWP/Hn0RhTv6lTVTMz615LOnRQ7dUr+FoD7loUtH9mplvelIA5GnBdDa1DmapWi8hY4DUgA3hSVReKyB1eQDNwk2c/LiLX4yqOR3sBN8o77wSvy811j2bx8rxps3v1Sr5/QwwfPrxOx6yJEyfy0ksvAbB8+XK+/PJLsrKy6uzTt29fhgwZAsBRRx1FWVlZ4PEXLFjAr3/9ayorK9m0aROnnnoqAG+99RZPP/00ABkZGXTr1o2nn36aUaNG0auXe9Dq2bNn4/44Y9JQaalrpbNsmbuGFBe7O/KvvoKrr06849++HTIy4He/g/vug9WrE48Ze8cP/sdvLqHWEajqTFU9RFUPUtVib9mtXhJAVT9V1e+p6mBVHaKqr4cZD7gTHHZ5W+fOnXd9fuedd3jzzTf54IMPmDdvHkOHDvXtuNWhQ4ddnzMyMpLWL4wePZqHHnqITz75hNtuu806ghnTBIKKd/zK+K+4whXpHHywK/7xs3UrjB8PEybUf80pKICyMqipce/NmQQg+sriZhdGeVvXrl3ZuHGj77r169fTo0cPMjMz+fzzz/nwww/3/oc8GzduZP/996eqqorSmFruH/zgBzzyyCMA7Ny5k/Xr13PiiSfyhz/8gbVejdS6desa/fvGtDZBjUgeegiuvz7xjn/rVti0CR54AA44wP+YTVrGH3LX4lYx1tCeKiho2oyblZXF9773PQYOHEinTp3o3bv3rnUjR47k0Ucf5fDDD+fQQw/lmGOOafTv3XnnnYwYMYLs7GxGjBixKwk98MADFBYW8sQTT5CRkcEjjzzCscceS1FREccffzwZGRkMHTqUyZMnNzoGY1qToiL/RiRX+zZodzZvhmuugawslzRi9/e749/ra05tlqr9gdosVXvgphBUeZCqr4ZUFpu9Y+fRtGZTp6rm5blK2rw8933DBtUXXkheqdu7t//yvLzkx673xxsqN7f+ABqAJJXFaVc0ZIxJP0Ft+Xv0gJ/8xJW4+MnLg3vvbUAZP6WUkU8NbSgjnwJiim4a03lpwYLgLsRN2LXYEkEKu+qqqxgyZEid11NPPRV1WMakJL9idFVYsgSuu86/LX9mJrz7LkyeHHyxr7eMP+hCP3Wq6x7s9+P1DRa0fTvcdhsceWRwlmrKrsVBjwqp+rKiofDYeTSpLFnpil9b/E6dVPfZJ3mxT21bflXV/xszVZdn5OlORJdn5On/jWlg8U1eXvDBk/04qL7xhmp1dd0/rndv1QMOcOsLClQfeaRJOhqQpGgo8gv7nr4sEYTHzqNJVUGdru65R3XyZNUuXfyvs126qD78sOr++/uv31XMXl+vLr8stG2b6ptvJr/Q/8//BP94baLo1k21bdvEdTfeWPcE7G0dg8cSgWkQO48mVQXddMdePy9kqi7B3dEvIU8vZGqd3ruj29VdP7rdVHc9XbtWNTvb/wd69FC98krXVTh2eUaGavv2yZNAbZYJSjJPPulqqjt1qidLNQ1LBKZB7DyaKPnd9K5erTplSvC1VkR14ULVq3tO1U3UvdhuIlOvztp9R1/Vvu76nW3aqubkJL+YJ3t16aI6Y4bqE0/UX3ST7I4+qAgpttyqCVgiMA1i59GELeh6GHTTfNpp7nObNv7Xytqb5o1Zeb4bbO/YVfWii1Q7dvQ/QMeOqnfeGdxGNCenYRfqxhTdBD3u2BNByImgCcrbGqNz587N+nsNZYnAhCmoQveGG1wxuV/RzgEHqM6Zo/rMM0luumtqklfMfuc7wesaMvJb2BfqZhp1zhJBrOYa6i8JSwSmtUp2jxXULwpcEvAr2rmI3QeIb9Xz/s+fUH30UdWBA4MPXHuxbsjFfE8fV5rymtEMN6fplQiuvdZ/HOraV3ylT+2rQ4fgfa69NukJvummm/Shhx7a9f22227TO++8U0888UQdOnSoDhw4UF9++eVd65Mlgo0bNwbuN2XKFD3iiCN00KBBevHFF6uq6qpVq/Tss8/WQYMG6aBBg/Rvf/tb0liTsURgGiPoevnjH6uOHLn7gu9Xobs8I8/33+XqNr1V585Vvf/+4ErVoUNVr7gicX18q5/GXMwjLkVoCpYIYl9Bdw6w14ngn//8px533HG7vh9++OG6bNkyXb9+vaqqVlRU6EEHHaQ1NTWqmjwRVFVV+e4XNK+A3xwEe8sSgWmM3Fz/C32bNu6m/WcdEu/6t9JB/9zpXK1J9u8y2at3b1c0pFr/xboVXMwbI70SQX1CKu877LDDdMWKFTp37lz97ne/qzt27NCrrrpKjzjiCB08eLB27NhRV65cqarJE0HQfhMnTtRbbrklYftevXrptm3bGhV7LUsEpj5+na5eeUX14ov9i3c200lv4G7V997TrV39Z2qpAdV27fz/Xe67r+of/xicCJq4ZU1rliwRpN8QEyFNSDBq1ChefPFFnn/+ec4//3xKS0upqKjgo48+Yu7cufTu3btB8wbs7X7GNIVkox3PurKUoY8UkrNzKW1QcnYuZdgjl7Pz8p8z4I+/oYRf0Jm6QylkspX/5ldw3HF03LjG9zdFBJ56yv/f5YQJbgb32pmj4jXlMAtpLNREICIjReQLEVkkIuN91t8nInO9179EpDLMeIDQJgA9//zzmTZtGi+++CKjRo1i/fr17LvvvrRr1463336bpX7TovkI2i9oXgG/OQiMCZLsQl9aCm/+rJR3luZTrW14Z2k+b/6slN//Hn7/H4s5/JFrEi70HdnOj1Y9wU3bfkNnNvn+pgK8/nryyXnr+3fZHDNKpbOgR4XGvnDTU34FfIfdcxb3T7L91bjpLMMtGgrRwIED9YQTTlBVV75/zDHH6MCBA3X06NF62GGH6ZIlS1Q1edFQsv0mT56sAwYM0EGDBumll16qqq6y+Mwzz9SBAwfq4MGD9f3339/r+FPlPJpwJKsvXb9edcw+iUU7O8jQ1SSffHcnorpjR/3FrlZhGymiqCMAjgVei/l+M3Bzku3fB06u77ipnAhaOjuPrVtenn9l7v77q3ZnnX6D/zALm+mkO+57UFdm+I+Zszwjz/1AQy70djGPTLJEEGbR0IHA8pjv5d6yBCKSB/QF3gpYXygic0RkTkVFRZMHakxrMuvKUsrb5lMjbShvm8+sK0tZsQK+u7SUxykkH1fGn89SpjCaWSv7sk6y2Bf/f1sd2Ua768ayqPAeNlO3eGYzmZQVesUzDSl2jXpyXuMrVSqLLwBeVNWdfitVtURVh6nqsOzs7GYOLRyffPJJwlwDI0aMiDos0xIkKegPqtB9r+9PeZQxCWX87ajmQFYit93G1m698bMly1XIfv/hAj4eU0J5Rh41COUZeXw8poTvP2wX+hYv6FGhsS/2oGgI+Bj4bkOOG1Q0VNtG3+ydmpoaKxpKFfUMvB8/eFpVu46qV12lW/7zLt1IwHjMtc00fZdL8LHbN2+vexMekhQNhTl5/Wygn4j0BVbg7vovit9IRA4DegAf7O0PdezYkbVr15KVleWaopk9oqqsXbuWjh07Rh2KKS2l+rJC2u7YPVF59WWFtN26FQ49lO3/cTUddtS9q29btQ0mTaITXgsdHzUIbXL7+E5vKHleE8yCAndBKCpy2+Xm0rZ2ii7TqolLFCEdXOR04H5cC6InVbVYRO7AZaYZ3ja3Ax1VNaF5qZ9hw4bpnDlz6iyrqqqivLzc2ts3QseOHcnJyaFdu3ZRh9L6lZbWudgSc7Hd1CufLmsb1tQ4Vg3ChNs2cP6dA+lTk7h/eUYeOVOK3RSKsdMmZmY2SfNpk/pE5CNVHea7MuhRIVVffkVDxrQYvsUvnVTHjdOaG36ZpPgGPZ1XdBkH+q5fQp6qup6/foO37Zp20VrtpC2SFA2F+kQQBr8nAmNaimR3/NtpjyJ0ZHvCujLy2HdzGeNzS/nd2sI6lb6byeTmrBImrnF39bOuLCW/pIgDdi7j64xcygqL61bomrSU7IkgVVoNGdOy1NNyJ775JsDOLxbROSAJ1CCMu2w913R+wreJ5oSsYjIzYcQDBYxtV0IZruVOGXmMbVfCiAd2X+i//3ABOdVltNEacqrLLAmY+gU9KqTqy4qGTOSStK7xK5rZRnut3O/QXUU8yYp2ks6tu/vnrXTH7DFa++ijxjQ5v6ttVZXqxx/rts49fS/mO9p10k109l9HW5193u/0V5kPJp9bN+CnjWmsZInA6giMiVdamti6pk0byMiAqqrA3Wr/Jfk1YK5BaKM1uwZ2u62qiFyWsYxcftOumJOeKrCGOyZUVkdgTDy/Mv5Nm+CVV2DMmLpJAKCmhp3tOzL55FJWcIDvIZeSR3kb/+GSv85wbfULCuCkpwo4Ia+MtlLDCXlllgRM5CwRmNapnvGWqy8rhKVLXeHM0qXUXPJT6NYNzjgD3bjR95BtNm/iylkXcUvbuwMrdJf+R3Hy8XiwURhM6rFEYFqf2qKdmAs9hYVw//3w/PPs+PmY3T13PW20hh3tu/DCFW+wnD6+h12Rkcu6dXDK5OCWOw0aj8eYVBNUeZCqL6ssNqqavEY1aFz8esbc2Yloz57+Uy5uIlMvwip0TctFRGMNGROO+Mrc2jv+FSugfXv3PcAb//URh4w/hzwSx9xZRi5LlsCgQQVcsRTuYneF7i0U837e7rv6ggIr0jGthxUNmdQVVM5fVJRYmbtlC9x0E1x/PTVt/O9vNmXl8eznR1IkdwWW8e+zjxv6Z3pmAX0pI4Ma+lLG9MwCmxXRtFqWCExq8ivnv+IKuP76pHf86/5Zxi86Tfa90N9CMQ8+CKc9k7x3bkjTWhuTuoLKjFL1ZXUErchelvPXIL7LN2bl6Usvua9+UzKKNOynjWmNsA5lJuX4ddrq0AHOOAOqq2H6dP/9RLh2n6e4a/2VvgOv3VVWwCGHwMqVibvm5bnmmsakI+tQZlLPzTcnlvNv3w5//CN89hlVHTJ9d9vUM5eJ6y/lCuoW7VxBCQ+tK6BLF7jnHjfMfqzMTKyM35gAoSYCERkpIl+IyCIR8Z14RkR+IiKfishCEXk2zHhMM4uv7H3iCXjuOTjnHFi+3H8fEfjiC8Z1Lgks5+/VC56jbmXucxSQu3uiLSvjN2ZPBJUZNfaFm5XsK+A7QHtgHtA/bpt+uPmKe3jf963vuFZH0EJMnaqaWbct/q7XAQfojo5dA8v5i4qSl/P7HTrTptY1JimS1BGE+UQwHFikqotVdQcwDTgrbpsrgEmq+q2XlFaHGI9pTr/6VWLRD0Dv3rB8OTd0fiTwjn/wYNhnn+C7frvjN6ZphZkIDgRin//LvWWxDgEOEZG/iciHIjIyxHhMU4sv+nnmGVfJe/LJ8PXX/vusXs3W7W14cG1BYDn/qFHw8MPJy/ltvB5jmk7UPYvb4oqHTgBygPdE5AhVrYzdSEQKgUKA3NqCYBMtv969l17qSmpycqB7d6isTNwvN5dVq9zH5yjgOepewfNiyvkhcI53Y0wTCvOJYAXUGb0rx1sWqxyYoapVqroE+BcuMdShqiWqOkxVh2VnZ4cWsImTbATPm25KLPpRhV69YMkSZl34kG/Rz6zTi+nbF+66q/6WPXbXb0wzCao8aOwLd7e/GOjL7sriAXHbjASmeJ974YqSspId1yqLm4lfjWyHDqpnnKE6aFBgZ6/aXlv77+9f2ZubW/cnrFOXMc2DqDqUicjpwP24FkRPqmqxiNzhBTRDRAS410sIO4FiVZ2W7JjWoayJlZb6l7/k5bllfv7939nx949pv6UyYdWmrDy6rClD/KbpwlXu1tQ0XfjGmIZJ1qHMehanM7/eve3awcEHw2ef+e/jXcmv6VXK79YWJvTuHdelhMc2FtCnD5SXJ+5uvXuNiYb1LE5nQeX8W7bAuHGJ5fxVVbBoEXTt6ns47eNqcx9a59/q5/HNriD/v/7Levca01JYImjN/Ebw/NnPoH9/6NEDVgd026iuZtbF/u38f9PBXcn79LHevca0FpYIWjO/cftr7/ivuQb23dd/v9xcLp4Z3M4frNWPMa2JJYLWLKiyt7oa7rmHWedN8L3r/7/Tilm2zP+Of906t53d8RvTelgiaI22b3cTuAQ1BPDKb4Lu+s+fvruIJ2BXwO74jWktou5ZbJrap5/CRRfBvHluqIdZs2Dr1t3rY8pvli2DpT69e2WVGy0ivkGRVfYa0zrZE0FLF9sqKCsLhgxxk7j/6U/w+uvMGv045Rnujr88I48HBpYw4Rt34e/Tx/+QNrCbMenF+hG0ZH79ANq0gYkT4aqrfFcDjBgBH37ov3tmpl3wjWmNrB9BazV+fOJVvqbGTdGFf6MhYNegb3bXb4wBSwSpz69D2BdfuFt5v667gC5bxrZtwY2GYpdbha8xxiqLU1nQUM87d7qJ3rt0gU2bEnZb3SGX6y9zZf1LlyYe1kbyNsbEsieCVOZXtrNzJ3TrBkuXMuuSR337AdxYVczw4a6Fjw3zYIypjyWCVBZUtrNhA/TuHdgP4K39CrjuOqsDMMY0jLUaSmX5+f5lO94Qnm3a+PcZs6GejTHxrNVQS3XrrYnLYsp2cnL8d7M6AGPMnrBEkMq++ca99+6dULazc6ebFjgjo+4uVgdgjNlToSYCERkpIl+IyCIRGe+zfrSIVIjIXO/18zDjaVHWrYPf/x5+9CPX8D+ufedNN8Enn8Do0VYHYIxpnNCaj4pIBjAJOBk3Sf1sEZmhqp/Gbfq8qo4NK44W63e/c5XCd92VsGrKFLj3Xhg7Fh58MILYjDGtSphPBMOBRaq6WFV3ANOAs0L8vdajvNxd4S+5BAYOrLPqww9d14ITT4QJEyKKzxjTqoSZCA4Elsd8L/eWxTtPROaLyIsi4jsMmogUisgcEZlTUVERRqyp5fbbXXOgO+5IWLV5MxxxBLzwgpte2BhjGivqyuI/AfmqOgh4A5jit5GqlqjqMFUdlp2d3awBNrvPPoOnnoIrr4S8vIQRJlatgtmz3UCjxhjTFMJMBCuA2Dv8HG/ZLqq6VlW3e1//BzgqxHhahqIi6NwZbrnFd8rhwkJ49tmogzTGtCZhJoLZQD8R6Ssi7YELgBmxG4jI/jFfzwQ+CzGe1Pf3v8NLL8EvfwnZ2b4jTGzZ4nKFMcY0ldBaDalqtYiMBV4DMoAnVXWhiNwBzFHVGcA1InImUA2sA0aHFU/KU3XDSu+7L4wbBzRs9FBjjGmsUEcfVdWZwMy4ZbfGfL4ZuDnMGFJeaam7xV+2zCWDn/7UjSqK6zm8fHniLtZz2BjTlKKuLE5v8ZUAAC++6JYDF1yQuIv1HDbGNDVLBFGqpxLg7rvhgQes57AxJlw2+miUAoYPVRHeeLWGU06JICZjTKtko4+mqoDC/oqOufzoR/D1180cjzEmLVkiiNI55yQsqm6fyXVbi7nzTjjggAhiMsakHUsEUfn2W5g2zT0V5OaCCFUH5FEoJXx9fAE33BB1gMaYdGGT10dl3DioqGDmb2Zz5eNDWQa0X+sqhT+fkjjPgDHGhMWeCKLw2msweTILfngTo+4auqv16PbtbtqBWbOiDtAYk04sETS3jRtd34HDDuO8uf+Z0Hp0xw4bQsIY07wsETSF+CFCvQ5hvsaPd92Fn3ySL5d39N3EhpAwxjQnqyNorNrewbW39rVDhEJiz6/33oOHH4brrqNq2LF06eIeEOLZEBLGmOZkTwSN1ZAhQktL3dX9+OOhbVs29j2CU091SaBtXCq2ISSMMc2tQU8EInIO8Jaqrve+dwdOUNWXwwuthQgqx1m61LUM2rqV6icm07Zqm1teXU37cVeTQwemTCkgI2P3mHO5uS4J2BASxpjm1KAhJkRkrqoOiVv2saoODSuwICk3xER+vrvox2vf3tUZbNvmu9v67nl0+7Ys1NCMMaZWUwwx4bed1S+Au4UXqbssMxOefBI2bqQG8d2ta6XVCBtjUkNDE8EcEZkgIgd5rwnAR2EG1mIcdpjrBNCzZ+IQoW3bsgz/mt+g5cYY09wamgiuBnYAzwPTgG3AVfXtJCIjReQLEVkkIuOTbHeeiKiI+D62pLTnnoN27eDLL11vsLKyOoX89/YsZjOZdXbZTCYTsqxG2BiTGhqUCFR1s6qOV9Vhqnq0qt6iqpuT7SMiGcAk4DSgP3ChiPT32a4rcC3w9z0PP2I1NfD883Dqqe6JwMfXJxRwBSWUkUcNQhl5jG1XwogHrEbYGJMaGpQIROQNr6VQ7fceIvJaPbsNBxap6mJV3YF7kjjLZ7s7gd/jnjJalvffh/Jy/6nEPFlZsOSYAk7IK6Ot1HBCXhknPVVgLYOMMSmjoRW+vVS1svaLqn4rIvvWs8+BQOyMu+XAiNgNRORIoI+qviIiNwYdSEQKgUKA3FTqbTVtGnTqBGf55TenpASqqxP7CxhjTKpoaB1BjYjsugKLSD7QqKnNRKQNMAGod8BlVS3xiqWGZWdnN+Znm051NfzhD3DGGbsmm4/16acwf777bEnAGJPKGnqJKgJmici7gAD/hneHnsQKoE/M9xxvWa2uwEDgHXHNL/cDZojImaqaQh0FArz9Nqxe7VsspApjxsC//uW6GLRvH0F8xhjTQA1KBKr6qteipxD4GHgZ2FrPbrOBfiLSF5cALgAuijnmeqBX7XcReQf4ZYtIAuCKhbp2hdNOS1j18stuWKFHH7UkYIxJfQ0dYuLnuJY9OcBc4BjgA+DEoH1UtVpExgKvARnAk6q6UETuAOao6oxGxh6d7dvhf//XTTXZqVPCqhtvhAED4PLLI4rPGGP2QEOLhq4FjgY+VNV/F5HDgLvq20lVZwIz45bdGrDtCQ2MJXqvvw6Vlb7FQpMmwVdfwauvWt2AMaZlaGhl8TZV3QYgIh1U9XPg0PDCSnHPPef6DZx0UsKqqio491zXtcAYY1qChiaCcq8fwcvAGyIyHfAZaS0NbN4M06fDj3/sehRTd16aRx5xicAYY1qKhlYWn+N9vF1E3ga6Aa+GFlUqe+UVN9/AhRcCezYvjTHGpKIGDUOdSiIfhvrcc+HDD910kxkZgaNQ5+W5YYeMMSYVNMUw1AZg/XqYORN+8hPIyACC56WxeYeNMS2FJYI98fLLrn1oTGuhoBEvUmkkDGOMScYSwZ6YNs3VCo/YPWRSoU//apt32BjTklgiaIjSUujTx3UOWLcOnn1216ojj4QDD4ScnMR5aYwxpiWwLk/1iW8WtGFDnWZBI0e6euP42SqNMaalsCeC+hQV7U4CtbZsgaIi3n3XVRlYEjDGtGSWCOoT0PxHly3jpJPg9tubNxxjjGlqlgjqs99+vou/7eKaBY0Z05zBGGNM07NEkMyOHbuGkYhV0ymT67cWc8kl1kzUGNPyWSJI5tZbXdHQuHGuOZDXLOiFk0p4ZmcB48dHHaAxxjSeJYIg774Ld9/tWgjde68bL6KmBl1SxmMbCxg1Cg45JOogjTGm8az5qJ/KSvjpT+Hgg2HChDqrROCtt2DTpmhCM8aYphbqE4GIjBSRL0RkkYgkFKSIyC9E5BMRmSsis0Skf5jxNNjYsbBiBUydCp0771q8ZYvrTybiZqk0xpjWILREICIZwCTgNKA/cKHPhf5ZVT1CVYcAdwMTiNpzz7lOZLfdBsOH11n12GOuqqC8PKLYjDEmBGE+EQwHFqnqYlXdAUwDzordQFU3xHztDEQzJnbszDIFBa5I6Oab66zOy3N1xlVVrvrAGGNaizATwYHA8pjv5d6yOkTkKhH5CvdEcI3fgUSkUETmiMicioqKpo2ydgiJpUtB1b3Ky+H55+usru1Xtn27+15a2rRhGGNMVEKbmEZEfgyMVNWfe98vAUao6tiA7S8CTlXVS5Mdt8knpqlnZhmbeMYY0xpENTHNCqBPzPccb1mQacDZIcbjr56ZZWziGWNMaxdmIpgN9BORviLSHrgAmBG7gYj0i/n6Q+DLEOPxV8/MMjbxjDGmtQstEahqNTAWeA34DHhBVReKyB0icqa32VgRWSgic4FxQNJioVAUF0OnTnWXxcwsU1zsvgasNsaYFs8mrwe44QbXcUzE3eoXF++aWWbVKjdffVmZ+xy32hhjWoRkdQTWsxigRw/3vn59Qk+xP/0JPvgA5s+HI46IIDZjjAmZjTUEMG8eHHSQb3fhl1+Gvn1h4MDmD8sYY5qDJQJwiWDw4ITFmzbBX/8KZ51ls5AZY1ovSwSbN8OiRb6J4LXXXAeys87y2c8YY1oJSwQLFrjexIMGJayqqoKjj4bvfz+CuIwxpplYIpg3z737PBFccAH84x/Q1qrUjTGtmCWC+fNdJXF+fp3F69e7JwJjjGntLBHMm+eKheJqg2+/3Y0nVF0dTVjGGNNc0jsRqLongrhiIVWYPh2GDrViIWNM65feiaCsDDZsSEgECxbAkiXWWsgYkx7SOxHMn+/e41oMTZ/uSorOPNNnH2OMaWXSOxHMm+eu+HFjR0yfDiNGwH77RRSXMcY0o/QuAZ83z01LGTNBPcB//7dVEhtj0kd6J4L582HIkITFxx/f/KEYY0xU0rdoaNMm+OqrhPqBxx6D2bMjiskYYyKQvongk09cO9GYFkMbNsDVV8MLL0QYlzHGNLNQE4GIjBSRL0RkkYiM91k/TkQ+FZH5IvJXEckLM546alsMxSSCv/zF9Sa2ZqPGmHQSWiIQkQxgEnAa0B+4UET6x232MTBMVQcBLwJ3hxVPgnnzoFu3OpMPT58O2dlw7LHNFoUxxkQuzCeC4cAiVV2sqjuAaUCde21VfVtVt3hfPwRyQoynrpihJUpL3XASzz0HW7bAtGnNFoUxxkQuzERwILA85nu5tyzI5cBf/FaISKGIzBGRORUVFY2PrKbG1REMHkxpKRQWwrJlbtXmze57aWnjf8YYY1qClKgsFpGLgWHAPX7rVbVEVYep6rDs7OzG/2BZGWzcCIMHU1TkngJibdkCRUWN/xljjGkJwuxHsALoE/M9x1tWh4icBBQBx6vq9hDj2a12DoJBg3Y9CcQLWm6MMa1NmE8Es4F+ItJXRNoDFwAzYjcQkaHAY8CZqro6xFjqmjcP2rSBgQNj64rrCFpujDGtTWiJQFWrgbHAa8BnwAuqulBE7hCR2uHc7gG6AH8QkbkiMiPgcE1r/nzo1w8yMykuhk6d6q7OzITi4maJxBhjIhfqEBOqOhOYGbfs1pjPJ4X5+4HmzYOjjgKgoMB9veceN/5cbq5LAgUFkURmjDHNLv3GGtqwARYvhssu27Wod2/3vno19OoVUVzGGBORlGg11KwWLHDvMT2KFyxwycCSgDEmHaVfIqhtMRSTCPr0sWEljDHpK/2KhubNg+7dIWd3J+Y77oguHGOMiVr6PRHUTlYvArgBSI0xJp2lVyKoqdmdCDyvvOIGmqutOjDGmHSTXolg8WI3mFDMZDQLFsCaNa6ewBhj0lF6JQKfiuKFC111QbduEcVkjDERS69EMH++G1piwIBdixYurPPVGGPSTnolgnnz4NBDd40psXMnfPYZDBwYcVzGGBOh9EsEMfUDW7fC2LFw6qkRxmSMMRFLj0RQWuoGESorg9de2zXrTJcuboyhk0+ONjxjjIlS6+9QVjsFWe3sM5WV7juw+uQCunZNHH3UGGPSSet/IkgyBdm111r9gDHGtP4ngiRTkC3cB/r3b95wjDEm1YT6RCAiI0XkCxFZJCLjfdYfJyL/FJFqEflxKEEETDWmfXL5/HNrOmqMMaElAhHJACYBpwH9gQtFJP7+exkwGng2rDgoLnZTjsXKzOTrscVUVVkiMMaYMJ8IhgOLVHWxqu4ApgF1BntW1TJVnQ/UhBZFQQGUlEBenhtoLi8PSkp4P99NQWZ1BMaYdBdmHcGBwPKY7+XAiBB/L1hBQcLck0MXwX33wWGHRRKRMcakjBbRakhECkVkjojMqaioaJJjHnwwXHedNR01xpgwE8EKIHZMzxxv2R5T1RJVHaaqw7Kzs5skuLfegpUrm+RQxhjTooWZCGYD/USkr4i0By4AZoT4ew22fTuccgpMmhR1JMYYE73QEoGqVgNjgdeAz4AXVHWhiNwhImcCiMjRIlIOjAIeE5GFYcUT61//cgPOWUWxMcaE3KFMVWcCM+OW3RrzeTauyKhZ1c5GZk1HjTGmhVQWN7UFCyAjAw45JOpIjDEmemmZCBYudEmgQ4eoIzHGmOi1/rGGfNxzDzRRK1RjjGnx0jIR9OvnXsYYY9KwaGjpUnj0UXsiMMaYWmmXCN59F8aMgTVroo7EGGNSQ9olgoULoX17N8SEMcaYNEwECxbAoYdCu3ZRR2KMMakh7RLBwoXWkcwYY2KlVSLYvNlVFtvQEsYYs1taNR/t3Bm+/RZUo47EGGNSR1olAoDu3aOOwBhjUktaFQ1NmQJ33RV1FMYYk1rSKhE8+yz88Y9RR2GMMaklrRKBtRgyxphEaZEISkuhTx9YsQL+9Cf33RhjjBNqIhCRkSLyhYgsEpHxPus7iMjz3vq/i0h+U8dQWgqFhVBe7r5XVrrvlgyMMcYJLRGISAYwCTgN6A9cKCL94za7HPhWVQ8G7gN+39RxFBXBli11l23Z4pYbY4wJ94lgOLBIVRer6g5gGnBW3DZnAVO8zy8CPxARacogli3bs+XGGJNuwkwEBwLLY76Xe8t8t/Emu18PZMUfSEQKRWSOiMyp2MPxo3Nz92y5McakmxZRWayqJao6TFWHZWdn79G+xcWQmVl3WWamW26MMSbcRLAC6BPzPcdb5ruNiLQFugFrmzKIggIoKYG8PBBx7yUlbrkxxphwh5iYDfQTkb64C/4FwEVx28wALgU+AH4MvKXa9CMBFRTYhd8YY4KElghUtVpExgKvARnAk6q6UETuAOao6gzgCeAZEVkErMMlC2OMMc0o1EHnVHUmMDNu2a0xn7cBo8KMwRhjTHItorLYGGNMeCwRGGNMmrNEYIwxaU5CaKQTKhGpAJYGrO4FrGnGcPZUKsdnse0di23vWGx7pzGx5amqb0esFpcIkhGROao6LOo4gqRyfBbb3rHY9o7FtnfCis2KhowxJs1ZIjDGmDTX2hJBSdQB1COV47PY9o7Ftncstr0TSmytqo7AGGPMnmttTwTGGGP2kCUCY4xJc60mEdQ3P3KURKRMRD4RkbkiMifiWJ4UkdUisiBmWU8ReUNEvvTee6RQbLeLyArv3M0VkdMjiq2PiLwtIp+KyEIRudZbHvm5SxJb5OdORDqKyD9EZJ4X22+85X29ecoXefOWt0+h2CaLyJKY8zakuWOLiTFDRD4WkT9738M5b6ra4l+40U2/Ar4DtAfmAf2jjismvjKgV9RxeLEcBxwJLIhZdjcw3vs8Hvh9CsV2O/DLFDhv+wNHep+7Av/CzcUd+blLElvk5w4QoIv3uR3wd+AY4AXgAm/5o8CYFIptMvDjqP+f8+IaBzwL/Nn7Hsp5ay1PBA2ZH9kAqvoebsjvWLFzR08Bzm7OmGoFxJYSVHWlqv7T+7wR+Aw31Wrk5y5JbJFTZ5P3tZ33UuBE3DzlEN15C4otJYhIDvBD4H+870JI5621JIKGzI8cJQVeF5GPRKQw6mB89FbVld7nVUDvKIPxMVZE5ntFR5EUW8USkXxgKO4OMqXOXVxskALnzivemAusBt7APb1XqpunHCL89xofm6rWnrdi77zdJyIdoogNuB/4FVDjfc8ipPPWWhJBqvu+qh4JnAZcJSLHRR1QEHXPnClzVwQ8AhwEDAFWAvdGGYyIdAH+CFynqhti10V97nxiS4lzp6o7VXUIbrra4cBhUcThJz42ERkI3IyL8WigJ3BTc8clImcAq1X1o+b4vdaSCBoyP3JkVHWF974aeAn3jyGVfCMi+wN476sjjmcXVf3G+8daAzxOhOdORNrhLrSlqvq/3uKUOHd+saXSufPiqQTeBo4FunvzlEMK/HuNiW2kV9SmqrodeIpoztv3gDNFpAxX1H0i8AAhnbfWkgh2zY/s1aJfgJsPOXIi0llEutZ+Bk4BFiTfq9nVzh2N9z49wljqqL3Ies4honPnlc8+AXymqhNiVkV+7oJiS4VzJyLZItLd+9wJOBlXh/E2bp5yiO68+cX2eUxiF1wZfLOfN1W9WVVzVDUfdz17S1ULCOu8RV0r3lQv4HRca4mvgKKo44mJ6zu4VkzzgIVRxwY8hysmqMKVMV6OK3v8K/Al8CbQM4Viewb4BJiPu+juH1Fs38cV+8wH5nqv01Ph3CWJLfJzBwwCPvZiWADc6i3/DvAPYBHwB6BDCsX2lnfeFgBT8VoWRfUCTmB3q6FQzpsNMWGMMWmutRQNGWOM2UuWCIwxJs1ZIjDGmDRnicAYY9KcJQJjjElzlgiM8YjIzpgRJ+dKE45iKyL5saOqGpNK2ta/iTFpY6u64QaMSSv2RGBMPcTNJ3G3uDkl/iEiB3vL80XkLW9wsr+KSK63vLeIvOSNcz9PRL7rHSpDRB73xr5/3evNiohc480lMF9EpkX0Z5o0ZonAmN06xRUNnR+zbr2qHgE8hBsVEuBBYIqqDgJKgYne8onAu6o6GDe/wkJveT9gkqoOACqB87zl44Gh3nF+Ec6fZkww61lsjEdENqlqF5/lZcCJqrrYG9xtlapmicga3LANVd7ylaraS0QqgBx1g5bVHiMfN8xxP+/7TUA7Vf2tiLwKbAJeBl7W3WPkG9Ms7InAmIbRgM97YnvM553srqP7ITAJ9/QwO2Z0SWOahSUCYxrm/Jj3D7zP7+NGhgQoAP7P+/xXYAzsmvikW9BBRaQN0EdV38aNe98NSHgqMSZMdudhzG6dvNmqar2qqrVNSHuIyHzcXf2F3rKrgadE5EagAviZt/xaoERELsfd+Y/BjarqJwOY6iULASaqGxvfmGZjdQTG1MOrIximqmuijsWYMFjRkDHGpDl7IjDGmDRnTwTGGJPmLBEYY0yas0RgjDFpzhKBMcakOUsExhiT5v4fRWBtwbP6BBsAAAAASUVORK5CYII=\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 绘制训练曲线\n",
    "def plot_metric(df_history, metric):\n",
    "    plt.figure()\n",
    "\n",
    "    train_metrics = df_history[metric]\n",
    "    val_metrics = df_history['val_' + metric]  #\n",
    "\n",
    "    epochs = range(1, len(train_metrics) + 1)\n",
    "\n",
    "    plt.plot(epochs, train_metrics, 'bo--')\n",
    "    plt.plot(epochs, val_metrics, 'ro-')  #\n",
    "\n",
    "    plt.title('Training and validation ' + metric)\n",
    "    plt.xlabel(\"Epochs\")\n",
    "    plt.ylabel(metric)\n",
    "    plt.legend([\"train_\" + metric, 'val_' + metric])\n",
    "    plt.savefig(cwd + 'imgs/' + metric + '.png')  # 保存图片\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "plot_metric(df_history, 'loss')\n",
    "plot_metric(df_history, metric_name)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 17.评估\n",
    "这里的评估 没有额外的使用测试集测评，还是拿验证集测试的。另外可以对一个法语句子进行翻译，看看翻译的结果如何\n",
    "\n",
    "以下步骤用于评估：\n",
    "- 用法语分词器（tokenizer_pt）编码输入语句。此外，添加<start>和<end>标记，这样输入就与模型训练的内容相同。这是编码器输入。\n",
    "- 解码器输入为 <start> token id\n",
    "- 计算padding mask 和 look ahead mask\n",
    "- 解码器通过查看编码器输出和它自身的输出（自注意力）给出预测。\n",
    "- 选择最后一个词并计算它的 argmax。\n",
    "- 将预测的词concat到解码器输入，然后传递给解码器。\n",
    "- 在这种方法中，解码器根据它之前预测的words预测下一个。"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "checkpoint: /home/xijian/pycharm_projects/DeepLearning/src/blogs/machine_translation/pytorch/transformer/./save/040_0.76_ckpt.tar\n",
      "Loading model ...\n",
      "Model loaded ...\n"
     ]
    }
   ],
   "source": [
    "# 加载model\n",
    "checkpoint = save_dir + '040_0.76_ckpt.tar'\n",
    "print('checkpoint:', checkpoint)\n",
    "# ckpt = torch.load(checkpoint, map_location=device)  # dict  save 在 CPU 加载到GPU\n",
    "ckpt = torch.load(checkpoint)  # dict  save 在 GPU 加载到 GPU\n",
    "# print('ckpt', ckpt)\n",
    "transformer_sd = ckpt['net']\n",
    "# optimizer_sd = ckpt['opt'] # 不重新训练的话不需要\n",
    "# lr_scheduler_sd = ckpt['lr_scheduler']\n",
    "\n",
    "reload_model = Transformer(num_layers,\n",
    "                           d_model,\n",
    "                           num_heads,\n",
    "                           dff,\n",
    "                           input_vocab_size,\n",
    "                           target_vocab_size,\n",
    "                           pe_input=input_vocab_size,\n",
    "                           pe_target=target_vocab_size,\n",
    "                           rate=dropout_rate)\n",
    "\n",
    "reload_model = reload_model.to(device)\n",
    "if ngpu > 1:\n",
    "    reload_model = torch.nn.DataParallel(reload_model,  device_ids=list(range(ngpu))) # 设置并行执行  device_ids=[0,1]\n",
    "\n",
    "\n",
    "print('Loading model ...')\n",
    "if device.type == 'cuda' and ngpu > 1:\n",
    "   reload_model.module.load_state_dict(transformer_sd)\n",
    "else:\n",
    "   reload_model.load_state_dict(transformer_sd)\n",
    "print('Model loaded ...')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******** final test...\n",
      "******** Test: loss: 1.343, test_acc: 0.765\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/torchtext/data/batch.py:23: UserWarning: Batch class will be retired soon and moved to torchtext.legacy. Please see the most recent release notes for further information.\n",
      "  warnings.warn('{} class will be retired soon and moved to torchtext.legacy. Please see the most recent release notes for further information.'.format(self.__class__.__name__), UserWarning)\n"
     ]
    }
   ],
   "source": [
    "def test(model, dataloader):\n",
    "    # model.eval() # 设置为eval mode\n",
    "\n",
    "    test_loss_sum = 0.\n",
    "    test_metric_sum = 0.\n",
    "    for test_step, (inp, targ) in enumerate(dataloader, start=1):\n",
    "        # inp [64, 10] , targ [64, 10]\n",
    "        loss, metric = validate_step(model, inp, targ)\n",
    "        # print('*'*8, loss, metric)\n",
    "\n",
    "        test_loss_sum += loss\n",
    "        test_metric_sum += metric\n",
    "    # 打印\n",
    "    print('*' * 8,\n",
    "          'Test: loss: {:.3f}, {}: {:.3f}'.format(test_loss_sum / test_step, 'test_acc', test_metric_sum / test_step))\n",
    "\n",
    "\n",
    "# 在测试集上测试指标，这里使用val_dataloader模拟测试集\n",
    "print('*' * 8, 'final test...')\n",
    "test(reload_model, val_dataloader)\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3, 5, 251, 17, 365, 35, 492, 390, 4, 2]\n",
      "<start> je pars en vacances pour quelques jours . <end>\n",
      "<start> i tennis very forgetful me helping bed . <end>\n"
     ]
    }
   ],
   "source": [
    "def tokenizer_encode(tokenize, sentence, vocab):\n",
    "    # print(type(vocab)) # torchtext.vocab.Vocab\n",
    "    # print(len(vocab))\n",
    "    sentence = normalizeString(sentence)\n",
    "    # print(type(sentence)) # str\n",
    "    sentence = tokenize(sentence)  # list\n",
    "    sentence = ['<start>'] + sentence + ['<end>']\n",
    "    sentence_ids = [vocab.stoi[token] for token in sentence]\n",
    "    # print(sentence_ids, type(sentence_ids[0])) # int\n",
    "    return sentence_ids\n",
    "\n",
    "\n",
    "def tokenzier_decode(sentence_ids, vocab):\n",
    "    sentence = [vocab.itos[id] for id in sentence_ids if id<len(vocab)]\n",
    "    # print(sentence)\n",
    "    return \" \".join(sentence)\n",
    "\n",
    "# 只有一个句子，不需要加pad\n",
    "s = 'je pars en vacances pour quelques jours .'\n",
    "print(tokenizer_encode(tokenizer, s, SRC_TEXT.vocab))\n",
    "\n",
    "\n",
    "s_ids = [3, 5, 251, 17, 365, 35, 492, 390, 4, 2]\n",
    "print(tokenzier_decode(s_ids, SRC_TEXT.vocab))\n",
    "print(tokenzier_decode(s_ids, TARG_TEXT.vocab))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "real target: i m taking a couple of days off .\n",
      "pred_sentence: <start> i m going on a couple of days off .\n"
     ]
    }
   ],
   "source": [
    "# inp_sentence 一个法语句子，例如\"je pars en vacances pour quelques jours .\"\n",
    "def evaluate(model, inp_sentence):\n",
    "    model.eval()  # 设置eval mode\n",
    "\n",
    "    inp_sentence_ids = tokenizer_encode(tokenizer, inp_sentence, SRC_TEXT.vocab)  # 转化为索引\n",
    "    # print(tokenzier_decode(inp_sentence_ids, SRC_TEXT.vocab))\n",
    "    encoder_input = torch.tensor(inp_sentence_ids).unsqueeze(dim=0)  # =>[b=1, inp_seq_len=10]\n",
    "    # print(encoder_input.shape)\n",
    "\n",
    "    decoder_input = [TARG_TEXT.vocab.stoi['<start>']]\n",
    "    decoder_input = torch.tensor(decoder_input).unsqueeze(0)  # =>[b=1,seq_len=1]\n",
    "    # print(decoder_input.shape)\n",
    "\n",
    "    with torch.no_grad():\n",
    "        for i in range(MAX_LENGTH + 2):\n",
    "            enc_padding_mask, combined_mask, dec_padding_mask = create_mask(encoder_input.cpu(), decoder_input.cpu()) ################\n",
    "            # [b,1,1,inp_seq_len], [b,1,targ_seq_len,inp_seq_len], [b,1,1,inp_seq_len]\n",
    "\n",
    "            encoder_input = encoder_input.to(device)\n",
    "            decoder_input = decoder_input.to(device)\n",
    "            enc_padding_mask = enc_padding_mask.to(device)\n",
    "            combined_mask = combined_mask.to(device)\n",
    "            dec_padding_mask = dec_padding_mask.to(device)\n",
    "\n",
    "            # forward\n",
    "            predictions, attention_weights = model(encoder_input,\n",
    "                                                   decoder_input,\n",
    "                                                   enc_padding_mask,\n",
    "                                                   combined_mask,\n",
    "                                                   dec_padding_mask)\n",
    "            # [b=1, targ_seq_len, target_vocab_size]\n",
    "            # {'..block1': [b, num_heads, targ_seq_len, targ_seq_len],\n",
    "            #  '..block2': [b, num_heads, targ_seq_len, inp_seq_len], ...}\n",
    "\n",
    "            # 看最后一个词并计算它的 argmax\n",
    "            prediction = predictions[:, -1:, :]  # =>[b=1, 1, target_vocab_size]\n",
    "            prediction_id = torch.argmax(prediction, dim=-1)  # => [b=1, 1]\n",
    "            # print('prediction_id:', prediction_id, prediction_id.dtype) # torch.int64\n",
    "            if prediction_id.squeeze().item() == TARG_TEXT.vocab.stoi['<end>']:\n",
    "                return decoder_input.squeeze(dim=0), attention_weights\n",
    "\n",
    "            decoder_input = torch.cat([decoder_input, prediction_id],\n",
    "                                      dim=-1)  # [b=1,targ_seq_len=1]=>[b=1,targ_seq_len=2]\n",
    "            # decoder_input在逐渐变长\n",
    "\n",
    "    return decoder_input.squeeze(dim=0), attention_weights\n",
    "    # [targ_seq_len],\n",
    "    # {'..block1': [b, num_heads, targ_seq_len, targ_seq_len],\n",
    "    #  '..block2': [b, num_heads, targ_seq_len, inp_seq_len], ...}\n",
    "\n",
    "\n",
    "\n",
    "# s = 'je pars en vacances pour quelques jours .'\n",
    "# evaluate(s)\n",
    "\n",
    "s = 'je pars en vacances pour quelques jours .'\n",
    "s_targ = 'i m taking a couple of days off .'\n",
    "pred_result, attention_weights = evaluate(reload_model, s)\n",
    "pred_sentence = tokenzier_decode(pred_result, TARG_TEXT.vocab)\n",
    "print('real target:', s_targ)\n",
    "print('pred_sentence:', pred_sentence)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input: je pars en vacances pour quelques jours .\n",
      "target: i m taking a couple of days off .\n",
      "pred: <start> i m going on a couple of days off .\n",
      "\n",
      "input: je ne me panique pas .\n",
      "target: i m not panicking .\n",
      "pred: <start> i m not getting not getting involved .\n",
      "\n",
      "input: je recherche un assistant .\n",
      "target: i am looking for an assistant .\n",
      "pred: <start> i m looking for a lot .\n",
      "\n",
      "input: je suis loin de chez moi .\n",
      "target: i m a long way from home .\n",
      "pred: <start> i m at home .\n",
      "\n",
      "input: vous etes en retard .\n",
      "target: you re very late .\n",
      "pred: <start> you re late .\n",
      "\n",
      "input: j ai soif .\n",
      "target: i am thirsty .\n",
      "pred: <start> i m thirsty .\n",
      "\n",
      "input: je suis fou de vous .\n",
      "target: i m crazy about you .\n",
      "pred: <start> i m crazy about you .\n",
      "\n",
      "input: vous etes vilain .\n",
      "target: you are naughty .\n",
      "pred: <start> you re foolish .\n",
      "\n",
      "input: il est vieux et laid .\n",
      "target: he s old and ugly .\n",
      "pred: <start> he s old and old and old .\n",
      "\n",
      "input: je suis terrifiee .\n",
      "target: i m terrified .\n",
      "pred: <start> i m being promoted .\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 批量翻译\n",
    "sentence_pairs = [\n",
    "    ['je pars en vacances pour quelques jours .', 'i m taking a couple of days off .'],\n",
    "    ['je ne me panique pas .', 'i m not panicking .'],\n",
    "    ['je recherche un assistant .', 'i am looking for an assistant .'],\n",
    "    ['je suis loin de chez moi .', 'i m a long way from home .'],\n",
    "    ['vous etes en retard .', 'you re very late .'],\n",
    "    ['j ai soif .', 'i am thirsty .'],\n",
    "    ['je suis fou de vous .', 'i m crazy about you .'],\n",
    "    ['vous etes vilain .', 'you are naughty .'],\n",
    "    ['il est vieux et laid .', 'he s old and ugly .'],\n",
    "    ['je suis terrifiee .', 'i m terrified .'],\n",
    "]\n",
    "\n",
    "\n",
    "def batch_translate(sentence_pairs):\n",
    "    for pair in sentence_pairs:\n",
    "        print('input:', pair[0])\n",
    "        print('target:', pair[1])\n",
    "        pred_result, _ = evaluate(reload_model, pair[0])\n",
    "        pred_sentence = tokenzier_decode(pred_result, TARG_TEXT.vocab)\n",
    "        print('pred:', pred_sentence)\n",
    "        print('')\n",
    "\n",
    "batch_translate(sentence_pairs)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input: elle est toujours habillee en noir .\n",
      "target: she is always dressed in black .\n",
      "pred: <start> she is always on their minds .\n",
      "\n",
      "input: il devient chauve .\n",
      "target: he s going bald .\n",
      "pred: <start> he s going to get married .\n",
      "\n",
      "input: c est pour vous que je suis venu .\n",
      "target: you re the reason i came .\n",
      "pred: <start> i m worried about you .\n",
      "\n",
      "input: je suis desole de t avoir hurle dessus .\n",
      "target: i m sorry i yelled at you .\n",
      "pred: <start> i m sorry i hurt you .\n",
      "\n",
      "input: elle n est pas apte pour le poste .\n",
      "target: she isn t fit for the job .\n",
      "pred: <start> she is not doing it for the job .\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 随机选个句子翻译\n",
    "def evaluateRandomly(n=10):\n",
    "    for i in range(n):\n",
    "        pair = random.choice(pairs)\n",
    "        print('input:', pair[0])\n",
    "        print('target:', pair[1])\n",
    "        pred_result, attentions = evaluate(reload_model, pair[0])\n",
    "        pred_sentence = tokenzier_decode(pred_result, TARG_TEXT.vocab)\n",
    "        print('pred:', pred_sentence)\n",
    "        print('')\n",
    "\n",
    "\n",
    "evaluateRandomly(5)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 18.attention 的可视化"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "outputs": [],
   "source": [
    "# 可视化attenton 这里我们只展示...block2的attention，即[b, num_heads, targ_seq_len, inp_seq_len]\n",
    "# attention: {'decoder_layer{i + 1}_block1': [b, num_heads, targ_seq_len, targ_seq_len],\n",
    "#             'decoder_layer{i + 1}_block2': [b, num_heads, targ_seq_len, inp_seq_len], ...}\n",
    "# sentence: [seq_len]，例如：'je recherche un assistant .'\n",
    "# pred_result: [seq_len]，例如：'<start> i m looking for an assistant .'\n",
    "# layer: 字符串类型，表示模型decoder的N层decoder-layer的第几层的attention，形如'decoder_layer{i}_block1'或'decoder_layer{i}_block2'\n",
    "def plot_attention_weights(attention, sentence, pred_sentence, layer):\n",
    "    sentence = sentence.split()\n",
    "    pred_sentence = pred_sentence.split()\n",
    "\n",
    "    fig = plt.figure(figsize=(16, 8))\n",
    "\n",
    "    # block2 attention[layer] => [b=1, num_heads, targ_seq_len, inp_seq_len]\n",
    "    attention = torch.squeeze(attention[layer], dim=0) # => [num_heads, targ_seq_len, inp_seq_len]\n",
    "\n",
    "    for head in range(attention.shape[0]):\n",
    "        ax = fig.add_subplot(2, 4, head + 1)  # 111是单个整数编码的子绘图网格参数。例如，“111”表示“1×1网格，第一子图”，“234”表示“2×3网格，第四子图”\n",
    "\n",
    "        cax = ax.matshow(attention[head].cpu(), cmap='viridis')  # 绘制网格热图？注意力权重\n",
    "        # fig.colorbar(cax)#给子图添加colorbar（颜色条或渐变色条）\n",
    "\n",
    "        fontdict = {'fontsize': 10}\n",
    "\n",
    "        # 设置轴刻度线\n",
    "        ax.set_xticks(range(len(sentence)+2))  # 算上start和end\n",
    "        ax.set_yticks(range(len(pred_sentence)))\n",
    "\n",
    "        ax.set_ylim(len(pred_sentence) - 1.5, -0.5)  # 设定y座标轴的范围\n",
    "\n",
    "        # 设置轴\n",
    "        ax.set_xticklabels(['<start>']+sentence+['<end>'], fontdict=fontdict, rotation=90)  # 顺时间旋转90度\n",
    "        ax.set_yticklabels(pred_sentence, fontdict=fontdict)\n",
    "\n",
    "        ax.set_xlabel('Head {}'.format(head + 1))\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "def translate(sentence_pair, plot=None):\n",
    "    print('input:', sentence_pair[0])\n",
    "    print('target:', sentence_pair[1])\n",
    "    pred_result, attention_weights = evaluate(reload_model, sentence_pair[0])\n",
    "    print('attention_weights:', attention_weights.keys())\n",
    "    pred_sentence = tokenzier_decode(pred_result, TARG_TEXT.vocab)\n",
    "    print('pred:', pred_sentence)\n",
    "    print('')\n",
    "\n",
    "    if plot:\n",
    "        plot_attention_weights(attention_weights, sentence_pair[0], pred_sentence, plot)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input: je pars en vacances pour quelques jours .\n",
      "target: i m taking a couple of days off .\n",
      "attention_weights: dict_keys(['decoder_layer1_block1', 'decoder_layer1_block2', 'decoder_layer2_block1', 'decoder_layer2_block2', 'decoder_layer3_block1', 'decoder_layer3_block2', 'decoder_layer4_block1', 'decoder_layer4_block2'])\n",
      "pred: <start> i m going on a couple of days off .\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": "<Figure size 1152x576 with 8 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABGQAAAI8CAYAAAC3eX1lAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABwgElEQVR4nO3deZhkdXn3//fdPd3TM8zCNiDKJrigGNYRUdGoIcY9Im4J+vzAKI8xEZeQxZjEaPSJiUvcooZgRA1RUUGNGCUuCBJFdhBRo4IoguwwDLN2378/6ow2wwzTXfWtU+dUvV/X1dd0na7+1Le6qj59+p5TVZGZSJIkSZIkqT5jg16AJEmSJEnSqHEgI0mSJEmSVDMHMpIkSZIkSTVzICNJkiRJklQzBzKSJEmSJEk1cyAjSZIkSZJUMwcykiRJkiRJNXMgI0mSJEmSVDMHMupJRGwXEWPV5w+JiGdFxMSg1yVpONk5kupk50iqm70zWhzIqFfnAFMR8QDgLODFwCkDXdFmLDVpqDS+c8DekYaInSOpbo3vHTunHAcy6lVk5t3Ac4D3Z+bzgP0HvKbNNb7UJM1ZGzoH7B1pWNg5kurWht6xcwpxIKNeRUQ8GjgGOLPaNj7A9WxJG0pN0ty0oXPA3pGGhZ0jqW5t6B07pxAHMurVq4HXAWdk5pURsQ/w9cEu6V7aUGqS5ubVNL9zwN6RhsWrsXMk1evVNL937JxCIjMHvQYNgYhYXE1JGycifhP4E+C8zPyHqtRenZknDHhpkrrU5M4Be0caNnaOpLo1uXfsnHIcyKgn1WT0Q8CSzNwzIg4E/m9mvmLAS7uXJpeapLlpU+eAvSO1nZ0jqW5t6h07p3c+ZUm9ehfwO8AtAJl5GfD4QS5ocxHx6Ij4HvD96vSBEfH+AS9LUnfeRcM7B+wdaYi8CztHUr3eRcN7x84pZ2QHMtHx2Yh42KDX0naZ+bPNNk0PZCFb9y4aXmoaDfZOGS3oHLB31AB2Thl2jjQ3dk45Leidd2HnFDGyAxngycAjgZcOeiEt97OIeAyQETEREScCVw16UZtrQalpNNg7vWtF54C9o0awc3pn50hzZ+eU0YresXPKGOWBzB/QKYtnRsSCQS+mxV4O/BHwAOA64KDqdJO0otQ0Euyd3rWhc8DeUTPYOb2zc6S5s3PKaEPv2DmFjOSL+kbEzsA3MnP/6rluX8vMTw96XeqP6vZ+N3AkEMBZwKsy85aBLkwjxd4ZLfaOBs3OGS12jgbNzhktdk45o3qEzIuBj1effxgPq+taRHwkIrafdXqHiPi3AS7pXjLz5sw8JjN3zcxdMvNFo1YWEXFURCwZ9DpGnL1TQBs6B+wdO6cR7JwC7Jz2sHcGzs4ppA29Y+eU65xRPZTsJcBTADLzgojYLSL22MLz4LRtB2Tm7ZtOZOZtEXHwANdzLxHxYeBeh4Jl5ksGsJzaRcS+wGnAK4EPDng5o8zeKaPxnQOj3Tt2TmPYOWXYOS1g7zSCnVNO43vHzinXOSN3hEw1bXxfZl43a/OJwM6DWVHrjUXEDptORMSONG/Q9wXgzOrjq8Ay4K6BrqhexwH/QOcXpQbA3imqDZ0Do907ds6A2TlF2TntYO8MkJ1TXBt6x84p1DlNu2H7LjNvj4jvbrbtvyPisYNaU8u9A/hWRHyKzvMHnwu8ZbBLuqfM/Mzs0xHxceCbA1pOrSJiHHgesBJ4VEQcWL0tnWpk7xTV+M6B0e0dO6cZ7Jyi7JyGs3cGz84prvG9Y+eU65yRO0Km8t45btM2ZOZHgaOBXwI3AM/JzI8NdlXb9GBgl0EvoiZPA76dmauAf6Pz6vcaDHungJZ2DoxO79g5zWHnFGDntIK90wx2TiEt7R07p0sjdYRMRDwaeAywIiJeO+tLy4DxwaxqKHwfuI3q/hQRe2bmtYNd0q9FxCo6z3GM6t8bgD8f6KLq8wfAO6vPzwDeHBEnZub6Aa5ppNg7fdHozoGR7h07Z8DsnL6wc5rN3hkgO6dvGt07dk65zhmpgQwwCSyhc72Xztp+J51DwTRPEfFK4A10JrjT/PpBecAg1zVbZi7d9rmGT/V83u0z8xyAzFwbEZ8GngR8aZBrGzH2TkFt6BwYzd6xcxrDzinIzmk2e6cR7JzC2tA7dk65zonMe7048lCrnvN1WmYePei1DIOI+BHwqCa/zVlEHHJfX8/Mi+tai0aTvVNOGzoH7B0Nlp1Tjp0jbZudU1YbesfOKWfUjpAhM6cj4v6DXscQ+Rlwx6AXsQ3vBw4BLqczYT4AuBBYS2fa/KTBLa0/LMlmsXeKakPnwIj1jp3TLHZOUXZOQ9k7zWHnFNeG3rFzNtNt54zcQKZyaUR8HvgUsHrTxsw8fXBLaq2fAGdHxJnAuk0bM/OdW/+W2v0CeFlmXgEQEY8A/jYzez6MMiLGgCWZeWevWYW9o/p3is4rgF/GPcvy0QNa1yizd8poQ+fA6PWOndM8dk4Zdk4zOwfsnaaxc8ppQ+/YOYU6Z1QHMlPALdxzcpeAhTF/11Yfk9VHEz10U1kAZOZ3I+Jh3YZFxH8AL6fznM4LgGUR8e7MfFvvSy0jM58IEBGnA4dsXpYDXNoos3fKaEPnwIj1jp3TSHZOGXZOAzsH7J0GsnPKaUPv2DmFOmfkXkNGoyciPk5nUv/v1aZj6Exdf6/LvEsz86CIOIbOoXp/AVyUmY15oa1NIuLKzNx/W9sklTWqvWPnSIMxqp0D9o40CHZOuc4ZySNkImKKzttV7U9nmgtAZr5kYItqqYhYAfwZ9/5ZdvW8wepFwT6amceUWSEAxwF/CLyqOn0O8IEe8iYiYgJ4NvC+zNwQEU2dbF4eESdzz7K8fIDrGVn2Thkt6RwY3d6xcxrCzimjdOdUme7rlGXvNICdU05L9nXsnEKdM1ZkSe3zMeB+wO8A3wB2B1YNdEXbEBE7RETjJoTAqcD3gQcCbwSuoXOYWVcycxrYKyKKHZ6XmWsz858y86jq458yc20PkR+kcz23A86JiL3ovLVfEx0HXEmnLF8FfK/apvq1qnfsnN6McO/YOc1h55RRtHPAfZ0+sHeaoVWdA6PTO3ZOcUU7ZySfshQRl2TmwRFxeWYeUE3jzs3Mwwe9ttki4mzgWXSOZLoIuBE4LzNfO8h1zRYRF2XmoZt+ltW2CzLzkT1kfhR4GPB57vmiYPN6IauIuILOc1e3qJtD4KoXmXpuZp42a1sA45m5cb55Gh1t6B07p7fOqbLsHTWCnVNGPzqnynBfR0OlDZ0Do9s7dk5zjeRTloAN1b+3Vy/CcwOwywDXszXLM/POiHgpncPM3hARTTsEc9PP8vqIeDqdV9zescfMH1cfY8DSHnKe0eM67iUzZyLiz4DTZm1LoJFlERGPpfMiU3sx6/GemfsMak0jrA29Y+f01jkw4r1j5zSKnVNGPzoH3Ncpxt5pjDZ0Doxu79g5hZTunFEdyJwUETsAf0VnSrgE+OvBLmmLFkTEbsDzgdcPejFb8eaIWA78CfBeYBnwml4CM/ONJRaWmT/d9Hl12NuDM/MrEbGI3u77X4mIE4FPcs8J8609ZPbLh+jcHhfRedVyDU4besfO6ZG9Y+c0iJ1TRvHOAfd1CrN3mqENnQMj2jt2TlFFO2dUn7L0wMy8elvbBi0ingv8DfDNzHxFROwDvC0zjx7w0voqIr7OFg6Fy+5fyOplwPHAjpm5b0Q8GPhgZv5Wl3lbup9kE/8nJiLOz8xHDXodakfv2Dn31G3nVJkj2Tt2TnPYOc3mvk459k4ztKFzYHR7x84pp3TnjOpA5uLMPGSzbRdl5qGDWtPmovNq2Cdk5j8Nei33JSI+ArwqM2+vTu8AvCN7eEX1iJh9O0wBRwMbM/PPusy7FDgMOD8zD662XZGZv9HtGtsiIt4KjAOnA+s2bc/Miwe2qBHV9N6xc36l586pMi9lBHvHzmkOO6eMfnROleO+TiH2TjM0vXNgtHvHzimndOeM1FOWImI/Om8ftjwinjPrS8uY9ZZiTZCZ0xHxe0CjCwM4YFNZAGTmbRFxcC+BmXnRZpvOi4jv9BC5LjPXd14bCiJiAffxYlTbEhH/Z0vbM/Oj3Wb20abp7cpZ2xLo+n/+NT9t6R075x567RwY3d6xcwbMzimueOdUOe7rlGPvDFBbOgdGu3fsnKKKds5IDWSAh9J5IaLtgWfO2r4KeNkgFrQN50XE+7j3c+maNPEfi4gdMvM2gIjYkR7vV1XGr/KBQ4HlPUR+IyL+ElgUEb8NvAL4zx7yZr/C+RTwW8DFQOMKIzOfOOg1qFW9Y+eU6RwY0d6xcxrBzimreOfMyvnVZeC+TtfsnYFrU+fAiPaOnVNO6c4Z1acsPTozvzXodWxL9Vy/zWUvr2tQWjXN/EvgU0AAzwXekpkf6yHzajpTxqDz6tpXA2/KzG92mTcG/AHw5Crzy8DJWejOHxHbA5/IzKeUyCspInYF/h9w/8x8akQ8HHh0Zn5owEsbOW3oHTunTOdUmSPZO3ZOc9g5ZfSjc6pc93UKsXeaoQ2dA6PbO3ZOOaU7Z1QHMv8IvBlYA3wJOAB4TWb++0AX1lIRsT+waVL4tcz83iDXU7eImAC+m5kPHfRaNhcR/wV8GHh9Zh5YHU54ySg8v7Np7J1yRr1zoLm9Y+c0h51Tjp3T3M4Be6cp7JyyRr13RqlzRu0pS5s8OTP/LCKOAq4BngOcAzSuMKLz3vP7M+s5mJn5pi6zVtA5dHBv7vme6T29MF1mXhkRN21aY0TsmZnXdptXPQD/EHh8tels4F8yc0OXeZsmwveQXb5qd0T856y8ceBhwGndZNVg58w8LSJeB5CZGyPCt4QcjFb0jp0D9Ng5Veao9o6d0xwj1zlVXvHeKd05VYb7OuXYO83Qis4B93WqTWdj53SraOeM6kBmovr36cCnMvOOTS9I1CQR8UFgMZ3p6Ml0Dlfr5cWXPgecC3yFAu+ZDhARzwLeAdwfuBHYC7iKTsl16wN0bqP3V6dfXG17aZd5s19waQp4HrDjVs47F2+f9flG4KeZ+fMe8vppdUTsRFVwEXE4cMdglzSyGt87dk6xzoHR7R07pzlGsXOgcO/0qXPAfZ2S7J1maHzngPs62DkllO2czBy5D+CtwPeBS+jcMVfQecuuXjKPAI6rPl8BPLDAOi/f7N8lwLk95F3ah5/lZcBOdA7Tgk65fajXzLls6/EyLurx+3el8wJmzwB2Kf1zLXg9DwHOq0riPOCHdF65feBrG7WP0r1j57Src6rMoe8dO6c5H23Y1yndOVXGpYV/jsU7Z1PuXLb1eBlD3znVOu2dBny0oXOqHPd17mNbj5dh53TxMZJHyGTmX1TPc7wjO29/djfwu93mRcQb6EwJH0rn+WQTdA7Pe2yPS11T/Xt3RNwfuAXYrYe8L0TE0zLziz2ua7YNmXlLRIxFxFhmfj0i3tVj5nRE7JuZPwaIiH3oYeIcEYfMOjlG57bq+r4fEc8H3kbnUL8A3hsRf5qZn+42s18y8+KI+E06980AfpA9PA1D3SvZO3ZOszunyhjJ3rFzmqMl+zqlOwfK904/Ogfc1ynG3mmGlnQOuK9j5/SodOeM3EAmIhYDD87My2Zt3onedr6PAg6m89ZcZOYvImJpD3mbfCE6rzD9j8Cm944/uYe8VwGvi4j1wAY6d6DMzGU9ZN4eEUvoHKp3akTcyKy3kOvSnwJfj4ifVKf3Bo7rIe8d/Po5iRvpPK/1eT3kvR54ZGbeCL967uhXgEYVxmb39SurbXtGxHRmXjfY1Y2WPvSOndPszoER7B07pzlatK9TunOgfO/0o3PAfZ0i7J1maFHngPs6dk4P+tE5IzeQofNAOT0iDsjMTXfsk+m8tVi3xb0+MzMiNj2PbLsC64TOc+n+EHgc8C06D8oP9JC3HDiGzuF+b4qIPen9f6K+XuW+CnhR9XnXL8ZXOQ/4FzrvP387nbdR6+Vt9L7Ar9/mjerzZ2x6XmtmvnOeeWObyqJyC53JcNP0476u7pS+LeycZncOjGbv2DnN0ZZ9ndKdA+V7px+dA+7rlGLvNENbOgfc17FzelP8vt7EK9lX1eFEZwDPh85EC1iRmRf2EHtaRPwLsH1EvAz4Kr3/Dw/AR+i8eNN7gPcCDwc+2kPePwOHA79XnV4FvK+XBdIZ6p1F5/CypcAnM/OWHjM/CjwQ+Ds613sf4GM95B1Kp3h3o/PiWC+n89y/pdXHfH0pIr4cEcdGxLHAF4H/6mF9fdGn+7q60Ifbws5pdufACPaOndMcLdrXKd05UL53+tE54L5OEfZOM7Soc8B9HTunB/24r0f1wjQjJSL2A07KzMdHxF8Bd2bme3rM/G3gydXJL2fmVwqs83uZ+fBtbZtH3sWZeUhEXJKZB1fbLsvMAwus9QDgBcDRwM8z88geskpf73OAp2fmqur0UuDMzHz8fX/nfWY+h18/h/XczPxst1n91I/7urpT+rawc5rbOdX3j2Tv2DnN0YZ9nT499vrSOyU7p8pzX6cQe6cZ2tA5Vab7OvexbR55dk6h+/ooPmWJzPx+dDwEeCGdQ9bmLSK+mZlHRMQq7nnI1ssjYga4FXhbZr5/qyH37eKIODwzv11d3qOAXibNGyJinF+/RdcKYKaHvNluBG6gc3jZLj1mlb7euwLrZ51eX22bl/u4vY8vdHsXV+q+rt6VuC3snHtocufAiPaOndMcLdnX6cdjr1+9U7JzwH2dYuydZmhJ54D7OnZOj0p3zkgeIQMQnUOhXgJcl5m/t42zd3sZOwH/k5kP7fL7r6Lz6s3XVpv2BH5A54WTMjMPmGfeMXSmrIfQOVzvucBfZeanullflfkKOodsrQA+BZyWmd/rNq/KLH29X1+t8Yxq07PpHPr3972scwuX09PtvZXM+2XmDT1mHEuf7+uam37fFnZO15lFr3eVObK9Y+c0R9P3dfr02CvaO/3onCrXfZ1fZ7qvMySa3jnV97uv02Hn9JZxLIXu66M8kFkMXA8cXeLwt/u4nN0y8/ouv3ev+/p6Zv60i8z96LyYUwBfzcyrulnbrLy/p/Pgu7SXnM0y+3G9D+HX08tzMvOSbtY2h8vp+vbeSt6Zmfn0HjNqua9r2+q4LeycrjKLX+8qdyR7x85pjqbv6/TxsVesd/rROVWu+zq/znNfZ0g0vXOq73VfZwvsnHlnFLuvj+xARpIkSZIkaVBG7l2WJEmSJEmSBs2BjCRJkiRJUs1GfiATEceb2cy8Uc5swxrVvVG9v7Qhsw1rbEumndMco3p/GdXMNqyxTZmav7bctj72mpvZhjWWyhz5gQzQj+Ie1cw2rLEtmW1Yo7o3qveXNmS2YY1tybRzmmNU7y+jmtmGNbYpU/PXltvWx15zM9uwxiKZDmQkSZIkSZJqNrTvsjQxuV1OLdphm+fbsH41E5PbzSlzZiLmdL6Na1ezYGpumQvuXDen862fWcPk2KJtn3EeN+ecMxeMzy1v+m4mxxfP6bwzE3PL3LBhNRMTc/tZjm2YntP51m+8m8kFc1snMbfbfP3G1UwumMM6Z+Z2A83nZ8n0xrllzqxlcmxqm+dbM72K9TNr53bF9SuTY4ty0YJlczrvXB97D37YHXPKu+mWaVbsNLfH1A8vn9v9agPrmGDhnM47VwPNnOM9ekOuYyLmuMY59u1gr/fcH8obci0Tse2OYPEczsP8+nvV6l/cnJkr5nRm/cpkLMwptv0zns998CEH3D2n85XunXk9TuZ4v57zfRpgjvvDbejGNqxx3pmFb/O1uZr16b7OfE3GVC6KbXfOetYxOcfb9sG/sXpO55tP5/zvVcvndL45/y00Pce/MXItk3PtnDne++a6/w7A2Bz/Zpvr9QZyamJumetXMznHv6ljzbb//p3P9c6Nc7t95tM5MT7Hn2WuYTK2/bNcM7P1v68WzOmSWmhq0Q4c8phXFs1cfb+53SHnY8VZVxfNm+sdcl523r545Lr7z+0P1/lY+PO5/fE6H7mw7EMk1m4omgfArbcXjfvWbZ8pmjcqFi1YxmN2fWHRzDO//MWieQC/84CDi2e2wVx/sc5HznHAOkh9ud4H7Fc88yvf+pufFg8dAVNsx6PGn1w088tfvqhoHsDv3P+gonkxMVk0DyA39uH385D+p+c2zfGPwvko3WXf3vClonmjYlFsx+FTTyua+V9f/nbRPICnP7LsGmduu71oHkAsKP9neCxdUjxz3UPuVzxz8oqyv/Knb76laB7A+LK5DfXm6lt3fm6rX/MpS5IkSZIkSTVzICNJkiRJklQzBzKSJEmSJEk1cyAjSZIkSZJUs74MZCLi2RHx8C6+7wkR8Zh+rEnS8LJzJNXN3pFUJztHGk5dDWQiYjLiPt/z7NnAvAojIhYATwAeM2vbtt+3WtLQs3Mk1c3ekVQnO0caTfMayETEwyLiHcAPgIdU294aEd+LiMsj4u3VBPZZwNsi4tKI2DciXhYRF0TEZRHxmYhYXH3vKRHxwYg4HzgNeDnwmur7Hge8ICK+GxF/EhErSl5xSc1n50iqm70jqU52jjTatvkG6NWk9vnAH1SbPgz8bWauioidgKOA/TIzI2L7zLw9Ij4PfCEzP11l3J6Z/1p9/uYq671V3u7AYzJzOiL+FrgrM99efe3ciDgTOBY4JyKuBE4GzsrMmZ6vvaTGsXMk1c3ekVQnO0fSJtscyADXA5cDL83M72/2tTuAtcCHIuILwBe2kvGIqii2B5YAX571tU9l5vTWLjwzfwb8XfX9TwX+DbiQzpT4HiLieOB4gIVT22/ziklqpFZ2ztT40m1fM0lN1c7eYfG2r5mkJmpn59znM6okdWMuT1l6LnAdcHpE/E1E7LXpC5m5ETgM+DTwDOBLW8k4BfjjzPwN4I3A1Kyvrd7WAiLiMOD9wHvoHHr3ui2dLzNPysyVmblyYtLCkFqqlZ0zObZoW7GSmquVvTPBwm3FSmqmVnbOpJ0jFbfNI2Qy8yzgrOrwuRcBn4uIm4GXAjcDizPzixFxHvCT6ttWAbP/u3gpcH1ETADH0CmgLVkFLNt0IiKeDLwduIHOoXSvysz187h+klrGzpFUN3tHUp3sHEmbzOUpSwBk5i3Au4F3VxPVaTpF8LmImAICeG119k8A/xoRJ9CZAP81cD5wU/Xv1o7t/0/g0xHxu8ArgVuAZ2bmT+d7xSS1m50jqW72jqQ62TmS5jyQmS0zvzPr5GFb+Pp53PNt2T5QfWx+vmM3O/1D4IBu1iRpeNk5kupm70iqk50jjaZ5ve21JEmSJEmSeudARpIkSZIkqWYOZCRJkiRJkmrmQEaSJEmSJKlmDmQkSZIkSZJq1tW7LLVCwtiGLBq58xd/VDQPgMmJsnkrlpXNA1i/oXjk1I9uLJ6Zq+8unsnda4rG/fw/9iqaB/CAF64uGzg9UzZvVMzMkKvL3hZPeeCjiuYBxEEPKpq3cfuFRfMAFty5rnjmTYeW78alP9tYPHO7S64tGxhRNg/I7/tOqU2Ryxez9vGHFs18+qN2K5oHsGDvsv//N73T1t7dt3vj//uz4plMlN/N7se+ztiKnYvmbdizbB7A+K1lf7/GTyaL5o2MhZPEvmX3ZZ/y9H2L5gFc/9zlRfN2//Q1RfMANuy9S/HMietvL565avfy+3k7/bzs7bNgovDf0wDj42Xz7tr670GPkJEkSZIkSaqZAxlJkiRJkqSaOZCRJEmSJEmqmQMZSZIkSZKkmjmQkSRJkiRJqpkDGUmSJEmSpJq1ciATEf8z6DVIGh12jqS62TuS6mTnSIPRyoFMZj5m0GuQNDrsHEl1s3ck1cnOkQajlQOZiLhr0GuQNDrsHEl1s3ck1cnOkQajlQMZSZIkSZKkNhuqgUxEHB8RF0bEhRs2rB70ciQNudmdsz7XDHo5kkbAPfZ11ruvI6m/7rGvM333oJcjDZ2hGshk5kmZuTIzV05MbDfo5UgacrM7ZzIWDXo5kkbAPfZ1Jt3XkdRf99jXGV886OVIQ2eoBjKSJEmSJElt4EBGkiRJkiSpZq0cyGTmkkGvQdLosHMk1c3ekVQnO0cajFYOZCRJkiRJktrMgYwkSZIkSVLNHMhIkiRJkiTVzIGMJEmSJElSzRzISJIkSZIk1WzBoBfQL7HqbhacfWnRzJmJ8j+uL119ftG8p+x1WNE8gOjD9WavBxSP3LDnzsUzx755adG83Y76ftE8gJkoO1fNzKJ5oyKnp5m+/Y6yoWPjZfOA65+4vGjejj/YUDQP4PZ9lhXP3OF/1xXPnLjp7uKZ0zffUjQvZ3w8D7O4426mzrygaOb0gomieQA/fPshRfMecuLFRfOAvuyXrN99h+KZMV3+Mb3ggquK5sXPfl40D2C6cF5m+d8JoyDXrmP6yh8UzRybmiqaB/BH/3Ft0bzT//n+RfMAJibLdy1ry9+v79o9imfuvL7svuPG628omtcPmVu/zh4hI0mSJEmSVDMHMpIkSZIkSTVzICNJkiRJklQzBzKSJEmSJEk1cyAjSZIkSZJUMwcykiRJkiRJNXMgI0mSJEmSVDMHMpIkSZIkSTVr7EAmIvaOiO9HxCkR8cOIODUijoyI8yLifyPisEGvUdLwsHMk1c3ekVQnO0dqnsYOZCoPAt4B7Fd9/D5wBHAi8JcDXJek4WTnSKqbvSOpTnaO1CALBr2Abbg6M68AiIgrga9mZkbEFcDem585Io4HjgeYYnGd65Q0HOwcSXWzdyTVyc6RGqTpR8ism/X5zKzTM2xhmJSZJ2XmysxcOcHCOtYnabjYOZLqZu9IqpOdIzVI0wcykiRJkiRJQ8eBjCRJkiRJUs0a+xoymXkN8IhZp4/d2tckqVd2jqS62TuS6mTnSM3jETKSJEmSJEk1cyAjSZIkSZJUMwcykiRJkiRJNXMgI0mSJEmSVLPGvqhvr2LBOOPbLy+buXxZ0TyAg976iqJ5D9jpx0XzAJhaWDxy49Kp4pl37V5+nTvs96CiefnzG4rmAcT9dy2bd035n+PIiCiblzNl84Dtf7KxaN52V91UNA/grvvvVjxz8ro7imfe8KRdimfutvoBRfPyzlVF8wBm7lpdPJPp8pHqTm5YXzxzry+W7Z3cuKFoHgC33F48cqIP+0//+5fl958eckXhda5bVzavH3LQC2inGBtjbNHispmLFxXNA/iPE59eNG/xTj8tmgdw6+Hl93W2u758f+90Vdn+Bli7b9n9p9hnRdE8gMlLry6aF3eMb/VrHiEjSZIkSZJUMwcykiRJkiRJNXMgI0mSJEmSVDMHMpIkSZIkSTVzICNJkiRJklQzBzKSJEmSJEk1cyAjSZIkSZJUs4EOZCLiTRFx5CDXIGm02DuS6mTnSKqTnSO1y4JBXnhm/s0gL1/S6LF3JNXJzpFUJztHapfiR8hExF9HxA8i4psR8fGIODEiDoqIb0fE5RFxRkTsUJ33lIh4bvX5NRHxxoi4OCKuiIj9qu0rIuK/I+LKiDg5In4aETuXXrek9rJ3JNXJzpFUJztHGl5FBzIR8UjgaOBA4KnAyupLHwX+PDMPAK4A3rCViJsz8xDgA8CJ1bY3AF/LzP2BTwN73sflHx8RF0bEhetn1vZ8fSQ13yB7Z3bnbGBdkesjqdmatK9j70jDr0mdsz79+0oqrfQRMo8FPpeZazNzFfCfwHbA9pn5jeo8HwEev5XvP7369yJg7+rzI4BPAGTml4DbtnbhmXlSZq7MzJWTY1M9XRFJrTGw3pndORMs7PmKSGqFxuzr2DvSSGhM50yGf19JpTXtXZY2/VfPNAN+fRtJI8PekVQnO0dSnewcqcFKD2TOA54ZEVMRsQR4BrAauC0iHled58XAN7YWsJXM5wNExJOBHQquV1L72TuS6mTnSKqTnSMNsaJT0sy8ICI+D1wO/JLO8xnvAP4/4IMRsRj4CXDcPGLfCHw8Il4MfAu4AVhVct2S2svekVQnO0dSnewcabj147C1t2fm31blcA5wUWZeChy++Rkz89hZn+896/MLgSdUJ+8AficzN0bEo4FHZqavYidpNntHUp3sHEl1snOkIdWPgcxJEfFwYAr4SGZe3GPensBpETEGrAde1usCJQ0de0dSnewcSXWyc6QhVXwgk5m/Xzjvf4GDS2ZKGi72jqQ62TmS6mTnSMOrae+yJEmSJEmSNPQcyEiSJEmSJNVsaN+LPjdOM33bHUUzx3cq/45wu3677Auaz6y+u2geQCxZXDzzpkOXFM9cu2MUz1zys+2K5o1dVf4F7Md+Nl02cP2GsnmjJHPQK9imDYvLzuGzD51z28PK/xxXnHp98cwFa3YpnpkLxovmlf49CMBM4c5R1yKCsYULy2ZOlc0DWLd92d3NxXvvWTQPgA0bi0dOL5ksnrnL56eKZzJReJ1R/v97xwrfL2ON/yfdjZyZYWbNmqKZYzMzRfMAfnFE2c7Z56zbi+YBTN1W/nfp5M9vK555zTN3LZ6553+Vve4LvnZZ0TyA6Sx7v8z72HeyjSRJkiRJkmrmQEaSJEmSJKlmDmQkSZIkSZJq5kBGkiRJkiSpZg5kJEmSJEmSauZARpIkSZIkqWYOZCRJkiRJkmrmQEaSJEmSJKlmjRvIRMRrI+K71cerI2LviLgqIv41Iq6MiLMiYtGg1ylpONg5kupm70iqk50jNVejBjIRcShwHPAo4HDgZcAOwIOBf87M/YHbgaMHtUZJw8POkVQ3e0dSnewcqdkWDHoBmzkCOCMzVwNExOnA44CrM/PS6jwXAXtv6Zsj4njgeIApFvd7rZLaz86RVLdyvRPb9XutktrPfR2pwRp1hMx9WDfr82m2MkjKzJMyc2VmrpxgYT0rkzSM7BxJdZt370zaO5K6576O1ABNG8icCzw7IhZHxHbAUdU2SeoHO0dS3ewdSXWyc6QGa9RTljLz4og4BfhOtelk4LbBrUjSMLNzJNXN3pFUJztHarZGDWQAMvOdwDs32/yIWV9/e70rkjTM7BxJdbN3JNXJzpGaq2lPWZIkSZIkSRp6DmQkSZIkSZJq5kBGkiRJkiSpZg5kJEmSJEmSauZARpIkSZIkqWaNe5elonKmaFzceVfRPIDp+y8vmrdgu8VF8wCuPWq34pl7nHlz8cyfPWPn4pkTt60pmjcdUTQPIKeny+ZlFs0bGQGxoHCljo+XzQOWXLeuaN66/fcomgewYHX5x8nYip2KZ+545arimbndVNG8sUVl8wBy/YbimawvHzkKEsjpsvs6uWZt0TyA5V+8smje9Pryd5gfvPvA4pl7fqF4JDucf33xzCy9v9yH313u6zRI4Z9dP26LZT8um5cby//e27i4/HERsabsPh7A+Nry+2Q3HbKwaN79v1q2w+rmETKSJEmSJEk1cyAjSZIkSZJUMwcykiRJkiRJNXMgI0mSJEmSVDMHMpIkSZIkSTVzICNJkiRJklQzBzKSJEmSJEk1cyAjSZIkSZJUs9YMZCLisxFxUURcGRHHD3o9koafvSOpTnaOpDrZOdLgLRj0AubhJZl5a0QsAi6IiM9k5i2DXpSkoWbvSKqTnSOpTnaONGBtGsicEBFHVZ/vATwYuEdhVJPd4wGmWFzv6iQNo/vsHTtHUmHu60iqk50jDVgrBjIR8QTgSODRmXl3RJwNTG1+vsw8CTgJYFnsmDUuUdKQmUvv3KNzxuwcSd3ral9nbCd7R1JX/PtKaoa2vIbMcuC2qiz2Aw4f9IIkDT17R1Kd7BxJdbJzpAZoy0DmS8CCiLgKeCvw7QGvR9Lws3ck1cnOkVQnO0dqgFY8ZSkz1wFPHfQ6JI0Oe0dSnewcSXWyc6RmaMsRMpIkSZIkSUPDgYwkSZIkSVLNHMhIkiRJkiTVzIGMJEmSJElSzRzISJIkSZIk1awV77LUjVgwzvgOOxbNnNl5h6J5AJM33lU0L9euLZoHsPuXby2euXb3ZcUzd7piQ/HMDTsuLpo3PjlZNA/grmccVDRv5qvfKJo3MhJy48aymaXzgLU7lr0PLj3nf4vmAUwesl/xzLxrdfHMXz5jj+KZKy4uu87xzKJ5ALlhffFMdScWTjK2955FM6d/+OOieQBjCxeWDRwfL5sH7PenVxXPjD12K57JhvK/F2LpkqJ517zyoUXzAPb98M+L5sV1E0XzRkpE4biyeQAL7yz7uy/60Dnrtyt/XEQu26545oK7yt8+k3eUvX3Gdyr7Nz/AzKqyf6Ozbus/R4+QkSRJkiRJqpkDGUmSJEmSpJo5kJEkSZIkSaqZAxlJkiRJkqSaOZCRJEmSJEmqmQMZSZIkSZKkmjV2IBMR10TEzoNeh6TRYe9IqpOdI6lOdo7UPI0dyEiSJEmSJA2rngYyEfF/IuLyiLgsIj4WEXtHxNeqbV+NiD2r850SEc+d9X13Vf8+ISLOiYgzI+IHEfHBiLjXmiLiRRHxnYi4NCL+JSLGe1m3pPaydyTVyc6RVCc7RxotXQ9kImJ/4K+AJ2XmgcCrgPcCH8nMA4BTgffMIeow4JXAw4F9gedsdjkPA14APDYzDwKmgWO6Xbek9rJ3JNXJzpFUJztHGj29HCHzJOBTmXkzQGbeCjwa+I/q6x8DjphDzncy8yeZOQ18fAvf81vAocAFEXFpdXqfLQVFxPERcWFEXLh+Zu18r4+k5mtU78zunA2s6+b6SGq2RnUObLavs/Hu+V4fSc3W6M5xX0cqb0FNl7ORavhTHTI3Oetrudl5Nz8ddKbCr9vWhWTmScBJAMsnVmyeI2m09L13ZnfOstjRzpFGW/37Oot2s3ek0VV757ivI5XXyxEyXwOeFxE7AUTEjsD/AC+svn4McG71+TV0prAAzwImZuUcFhEPrIrkBcA3N7ucrwLPjYhdNl1OROzVw7oltZe9I6lOdo6kOtk50ojp+giZzLwyIt4CfCMipoFL6DxX8cMR8afATcBx1dn/FfhcRFwGfAlYPSvqAuB9wIOArwNnbHY534uIvwLOqkplA/BHwE+7XbukdrJ3JNXJzpFUJztHGj09PWUpMz8CfGSzzU/awvl+CRw+a9Ofz/r8zsx8xha+Z+9Zn38S+GQva5U0HOwdSXWycyTVyc6RRktPb3stSZIkSZKk+avrRX23KDPPBs4e5BokjRZ7R1Kd7BxJdbJzpHbxCBlJkiRJkqSaOZCRJEmSJEmqmQMZSZIkSZKkmg30NWT6Kaenmbn9jqKZ40u2K5oHMLNscdnA6ZmyeUBcf0vxzJuevGPxzPH1xSNZ9tMomrdo3bqieQDLvvGjonnjq9YWzVMPxsaLR950UNnMJWeVv0/f7/w1xTNzTfn79eo9snjmbv99V9G8mQ0bi+YBfblfMl0+chTkuvXMXPOzQS9j2/bYrWzetb8omwfkmvK9c+uhOxXPvPOBOxfP3OdD1xTN2/vvvlM0D2Bm4cKygRv70I3qytj2y4tnLrn27rKBUf4YhsU3lb8Prn3AsuKZM5PFI9mwpOzfV7mu/B+BWXr/Kbe+z+gRMpIkSZIkSTVzICNJkiRJklQzBzKSJEmSJEk1cyAjSZIkSZJUMwcykiRJkiRJNXMgI0mSJEmSVLPWDWQi4oSIuCoiTh30WiQNPztHUp3sHEl1s3ekwVkw6AV04RXAkZn580EvRNJIsHMk1cnOkVQ3e0cakEYfIRMRr42I71Yfr46IDwL7AP8VEa8Z9PokDRc7R1Kd7BxJdbN3pGZp7BEyEXEocBzwKCCA84EXAU8BnpiZN2/he44HjgeYYnF9i5XUenaOpDp10znV99k7krrivo7UPE0+QuYI4IzMXJ2ZdwGnA4+7r2/IzJMyc2VmrpyIhbUsUtLQ6K1zsHMkzcu8Owc239eZ6vsiJQ0V93WkhmnyQEaSJEmSJGkoNXkgcy7w7IhYHBHbAUdV2ySpH+wcSXWycyTVzd6RGqaxryGTmRdHxCnAd6pNJ2fmJRExwFVJGlZ2jqQ62TmS6mbvSM3T2IEMQGa+E3jnZtv2HsxqJA07O0dSnewcSXWzd6RmafJTliRJkiRJkoaSAxlJkiRJkqSaOZCRJEmSJEmqmQMZSZIkSZKkmjX6RX17kpAbN5bN7MMrkI/duqpoXo6Xn7HF4qnimbtcvK545k9eVP72WX5182eWMVX49onmX+eRMTNdPHJsfeG8HXcoGwh9+a+CsfvtUjxz++8XjyQXTRbNG1u+tGgewPSttxfPVLcSpgv3RGbZPIBf3lw0bmb13UXzAMaXLyuemePFI6EPb4aT2xfuiZtvKZsHxHjhH6ZvKtQYG395Y/HMDQfsUTRvMmeK5gHcfMBE8cw9P3Vd8cydl96veOYt+5d9POdD9iyaB8DF3yufuRX+5SVJkiRJklQzBzKSJEmSJEk1cyAjSZIkSZJUMwcykiRJkiRJNXMgI0mSJEmSVDMHMpIkSZIkSTWrfSATEX8bESfWfbmSRpOdI6lOdo6kutk7Unt5hIwkSZIkSVLNahnIRMTrI+KHEfFN4KHVtpdFxAURcVlEfCYiFkfE0oi4OiImqvMs23Q6Ik6IiO9FxOUR8Yk61i2pnewcSXWycyTVzd6RhkPfBzIRcSjwQuAg4GnAI6svnZ6Zj8zMA4GrgD/IzFXA2cDTq/O8sDrfBuAvgIMz8wDg5f1et6R2snMk1cnOkVQ3e0caHnUcIfM44IzMvDsz7wQ+X21/REScGxFXAMcA+1fbTwaOqz4/Dvhw9fnlwKkR8SJg45YuKCKOj4gLI+LCDazrx3WR1Hx2jqQ61dY5sFnvpL0jjSj3daQhMcjXkDkF+OPM/A3gjcAUQGaeB+wdEU8AxjPzu9X5nw78M3AIcEFELNg8MDNPysyVmblygoX9vwaS2uQU7BxJ9TmFwp1Tff+veyfsHUn3cAru60itUsdA5hzg2RGxKCKWAs+sti8Frq+ez3jMZt/zUeA/qKa3ETEG7JGZXwf+HFgOLKlh7ZLax86RVCc7R1Ld7B1pSGzxf19KysyLI+KTwGXAjcAF1Zf+GjgfuKn6d+msbzsVeDPw8er0OPDvEbEcCOA9mXl7v9cuqX3sHEl1snMk1c3ekYZH3wcyAJn5FuAtW/jSB7byLUcAn95UCtWLTh3Rn9VJGjZ2jqQ62TmS6mbvSMOhloHMfETEe4Gn0nnFcEnqKztHUp3sHEl1s3ek5mrcQCYzXznoNUgaHXaOpDrZOZLqZu9IzTXId1mSJEmSJEkaSQ5kJEmSJEmSauZARpIkSZIkqWaNew2ZYiKIicmymRs2ls0DZu64s2je2I7bF80DyMmJ4pkLry97vQH2OXVZ8cxrnzxeNG+fM8vmAdx52O5F86a/WvhxM0oiyuZlls0DllxXOLMPa1y7U/n74MTldxTPnLp9l+KZdz50edG87W9dVTQPgJlbymeqK7FggvFdy94PN173i6J5QPGeGN9px6J5ALl2bfHMHb9b/vG308V92BedLPvnwJ3PPrhoHsD2V9xaNvDH5fdtR0IEsaDszy6np4vmASz6wS+L5s1Mlt8v2eGHfXgsL11UPHPplTcXz1z0yyVF8ybfWX6/ZP2TC9/m67b+N4JHyEiSJEmSJNXMgYwkSZIkSVLNHMhIkiRJkiTVzIGMJEmSJElSzRzISJIkSZIk1cyBjCRJkiRJUs0aP5CJiBMi4qqIODUiFkbEVyLi0oh4waDXJmn42DmS6mbvSKqTnSM1x4JBL2AOXgEcmZk/j4jDATLzoMEuSdIQs3Mk1c3ekVQnO0dqiEYNZCLitcBLqpMnA/sB+wD/FRH/DrwMWBERlwJHZ+aPB7JQSUPBzpFUN3tHUp3sHKnZGjOQiYhDgeOARwEBnA+8CHgK8MTMvDkizgdOzMxnDG6lkoaBnSOpbvaOpDrZOVLzNWYgAxwBnJGZqwEi4nTgcfMJiIjjgeMBplhcfIGShoqdI6luZXtnfGnxBUoaKu7rSA3X+Bf1nY/MPCkzV2bmyomYGvRyJA25e3QOCwe9HEkjYHbvTI4tGvRyJA05/76S+qtJA5lzgWdHxOKI2A44qtomSf1g50iqm70jqU52jtRwjXnKUmZeHBGnAN+pNp2cmZdExABXJWlY2TmS6mbvSKqTnSM1X2MGMgCZ+U7gnZtt23vW52cDZ9e6KElDy86RVDd7R1Kd7Byp2Zr0lCVJkiRJkqSR4EBGkiRJkiSpZg5kJEmSJEmSauZARpIkSZIkqWYOZCRJkiRJkmoWmTnoNfRFRNwE/HQOZ90ZuLnwxY9qZhvW2JbMQa5xr8xcUfiyh948OgeG6/4ybJltWGNbMueTZ+90Ycj2ddqwxrZktmGNg860c7owZJ3Tj8w2rLEtmW1Y43wyt9o5QzuQmauIuDAzV5rZvLxRzmzDGtW9Ub2/tCGzDWtsS6ad0xyjen8Z1cw2rLFNmZq/tty2Pvaam9mGNZbK9ClLkiRJkiRJNXMgI0mSJEmSVDMHMnCSmY3NG+XMNqxR3RvV+0sbMtuwxrZk2jnNMar3l1HNbMMa25Sp+WvLbetjr7mZbVhjkcyRfw0ZzV9E3JWZS2adPhZYmZl/XCD7bODEzLxws+1/DLwa2BdYkZmlX5BJUkMNqHNOBVYCG4DvAP83Mzf0enmS2mFAvfMhOr0TwA+BYzPzrl4vT1LzDaJzZn39PcBLZl++6uMRMmqL84Ajmfu72EhSL04F9gN+A1gEvHSwy5E0Al6TmQdm5gHAtUDPf4hJ0n2JiJXADoNexyhzIKOiImJFRHwmIi6oPh5bbT8sIr4VEZdExP9ExEOr7Ysi4hMRcVVEnEHnD597ycxLMvOa+q6JpDboY+d8MSt0jpDZvbYrJanR+tg7d1bnj+o8HsYuqW+dExHjwNuAP6vtyuheFgx6AWqlRRFx6azTOwKfrz5/N/BPmfnNiNgT+DLwMOD7wOMyc2NEHAn8P+Bo4A+BuzPzYRFxAHBxXVdCUmsMrHMiYgJ4MfCqkldIUuMNpHci4sPA04DvAX9S+DpJaq5BdM4fA5/PzOs7c2ANggMZdWNNZh606cSm5zhWJ48EHj7rQb0sIpYAy4GPRMSD6fyPz0T19ccD7wHIzMsj4vK+r15S2wyyc94PnJOZ5xa4HpLaYyC9k5nHVf9r/V7gBcCHS10hSY1Wa+dExP2B5wFPKH1FND8OZFTaGHB4Zq6dvTEi3gd8PTOPioi9gbMHsDZJw6dvnRMRbwBWAP+3wDolDY++7utk5nREfILO0wgcyEjqR+ccDDwI+FE16FkcET/KzAeVWbLmyteQUWlnAa/cdCIiDqo+XQ5cV31+7KzznwP8fnXeRwAH9H2FkoZJXzonIl4K/A7we5k5U3TFktqueO9Ex4M2fQ48i87TESSpeOdk5pmZeb/M3Dsz96bzFCeHMQPgQEalnQCsjIjLI+J7wMur7f8I/H1EXMI9j8z6ALAkIq4C3gRctKXQiDghIn5O54U1L4+Ik/t2DSS1SV86B/ggsCvwrYi4NCL+pj/Ll9RC/eidoPPUgyuAK4DdqvNKUr/2ddQA0XkDCUmSJEmSJNXFI2QkSZIkSZJq5kBGkiRJkiSpZg5kJEmSJEmSauZARpIkSZIkqWYOZCRJkiRJkmrmQEaSJEmSJKlmDmQkSZIkSZJq5kBGkiRJkiSpZg5kJEmSJEmSauZARpIkSZIkqWYOZNSTiNguIsaqzx8SEc+KiIlBr0vScLJzJNXJzpFUN3tntDiQUa/OAaYi4gHAWcCLgVMGuqLNWGrSUGl854C9Iw0RO0dS3RrfO3ZOOQ5k1KvIzLuB5wDvz8znAfsPeE2ba3ypSZqzNnQO2DvSsLBzJNWtDb1j5xTiQEa9ioh4NHAMcGa1bXyA69mSNpSapLlpQ+eAvSMNCztHUt3a0Dt2TiEOZNSrVwOvA87IzCsjYh/g64Nd0r20odQkzc2raX7ngL0jDYtXY+dIqteraX7v2DmFRGYOeg0aAhGxuJqSNk5E/CbwJ8B5mfkPVam9OjNPGPDSJHWpyZ0D9o40bOwcSXVrcu/YOeU4kFFPqsnoh4AlmblnRBwI/N/MfMWAl3YvTS41SXPTps4Be0dqOztHUt3a1Dt2Tu98ypJ69S7gd4BbADLzMuDxg1zQ5iLi0RHxPeD71ekDI+L9A16WpO68i4Z3Dtg70hB5F3aOpHq9i4b3jp1TzsgOZKLjsxHxsEGvpe0y82ebbZoeyEK27l00vNQ0GuydMlrQOWDvqAHsnDLsHGlu7JxyWtA778LOKWJkBzLAk4FHAi8d9EJa7mcR8RggI2IiIk4Erhr0ojbXglLTaLB3eteKzgF7R41g5/TOzpHmzs4poxW9Y+eUMcoDmT+gUxbPjIgFg15Mi70c+CPgAcB1wEHV6SZpRalpJNg7vWtD54C9o2awc3pn50hzZ+eU0YbesXMKGckX9Y2InYFvZOb+1XPdvpaZnx70utQf1e39buBIIICzgFdl5i0DXZhGir0zWuwdDZqdM1rsHA2anTNa7JxyRvUImRcDH68+/zAeVte1iPhIRGw/6/QOEfFvA1zSvWTmzZl5TGbumpm7ZOaLRq0sIuKoiFgy6HWMOHungDZ0Dtg7dk4j2DkF2DntYe8MnJ1TSBt6x84p1zmjeijZS4CnAGTmBRGxW0TssYXnwWnbDsjM2zedyMzbIuLgAa7nXiLiw8C9DgXLzJcMYDm1i4h9gdOAVwIfHPByRpm9U0bjOwdGu3fsnMawc8qwc1rA3mkEO6ecxveOnVOuc0buCJlq2vi+zLxu1uYTgZ0Hs6LWG4uIHTadiIgdad6g7wvAmdXHV4FlwF0DXVG9jgP+gc4vSg2AvVNUGzoHRrt37JwBs3OKsnPawd4ZIDunuDb0jp1TqHOadsP2XWbeHhHf3Wzbf0fEYwe1ppZ7B/CtiPgUnecPPhd4y2CXdE+Z+ZnZpyPi48A3B7ScWkXEOPA8YCXwqIg4sHpbOtXI3imq8Z0Do9s7dk4z2DlF2TkNZ+8Mnp1TXON7x84p1zkjd4RM5b1z3KZtyMyPAkcDvwRuAJ6TmR8b7Kq26cHALoNeRE2eBnw7M1cB/0bn1e81GPZOAS3tHBid3rFzmsPOKcDOaQV7pxnsnEJa2jt2TpdG6giZiHg08BhgRUS8dtaXlgHjg1nVUPg+cBvV/Ski9szMawe7pF+LiFV0nuMY1b83AH8+0EXV5w+Ad1afnwG8OSJOzMz1A1zTSLF3+qLRnQMj3Tt2zoDZOX1h5zSbvTNAdk7fNLp37JxynTNSAxlgElhC53ovnbX9TjqHgmmeIuKVwBvoTHCn+fWD8oBBrmu2zFy67XMNn+r5vNtn5jkAmbk2Ij4NPAn40iDXNmLsnYLa0Dkwmr1j5zSGnVOQndNs9k4j2DmFtaF37JxynROZ93px5KFWPefrtMw8etBrGQYR8SPgUU1+m7OIOOS+vp6ZF9e1Fo0me6ecNnQO2DsaLDunHDtH2jY7p6w29I6dU86oHSFDZk5HxP0HvY4h8jPgjkEvYhveDxwCXE5nwnwAcCGwls60+UmDW1p/WJLNYu8U1YbOgRHrHTunWeycouychrJ3msPOKa4NvWPnbKbbzhm5gUzl0oj4PPApYPWmjZl5+uCW1Fo/Ac6OiDOBdZs2ZuY7t/4ttfsF8LLMvAIgIh4B/G1m9nwYZUSMAUsy885eswp7R/XvFJ1XAL+Me5blowe0rlFm75TRhs6B0esdO6d57Jwy7Jxmdg7YO01j55TTht6xcwp1zqgOZKaAW7jn5C4BC2P+rq0+JquPJnroprIAyMzvRsTDug2LiP8AXk7nOZ0XAMsi4t2Z+bbel1pGZj4RICJOBw7ZvCwHuLRRZu+U0YbOgRHrHTunkeycMuycBnYO2DsNZOeU04besXMKdc7IvYaMRk9EfJzOpP7fq03H0Jm6/l6XeZdm5kERcQydQ/X+ArgoMxvzQlubRMSVmbn/trZJKmtUe8fOkQZjVDsH7B1pEOyccp0zkkfIRMQUnber2p/ONBeAzHzJwBbVUhGxAvgz7v2z7Op5g9WLgn00M48ps0IAjgP+EHhVdfoc4AM95E1ExATwbOB9mbkhIpo62bw8Ik7mnmV5+QDXM7LsnTJa0jkwur1j5zSEnVNG6c6pMt3XKcveaQA7p5yW7OvYOYU6Z6zIktrnY8D9gN8BvgHsDqwa6Iq2ISJ2iIjGTQiBU4HvAw8E3ghcQ+cws65k5jSwV0QUOzwvM9dm5j9l5lHVxz9l5toeIj9I53puB5wTEXvReWu/JjoOuJJOWb4K+F61TfVrVe/YOb0Z4d6xc5rDzimjaOeA+zp9YO80Q6s6B0and+yc4op2zkg+ZSkiLsnMgyPi8sw8oJrGnZuZhw96bbNFxNnAs+gcyXQRcCNwXma+dpDrmi0iLsrMQzf9LKttF2TmI3vI/CjwMODz3PNFweb1QlYRcQWd565uUTeHwFUvMvXczDxt1rYAxjNz43zzNDra0Dt2Tm+dU2XZO2oEO6eMfnROleG+joZKGzoHRrd37JzmGsmnLAEbqn9vr16E5wZglwGuZ2uWZ+adEfFSOoeZvSEimnYI5qaf5fUR8XQ6r7i9Y4+ZP64+xoClPeQ8o8d13EtmzkTEnwGnzdqWQCPLIiIeS+dFpvZi1uM9M/cZ1JpGWBt6x87prXNgxHvHzmkUO6eMfnQOuK9TjL3TGG3oHBjd3rFzCindOaM6kDkpInYA/orOlHAJ8NeDXdIWLYiI3YDnA68f9GK24s0RsRz4E+C9wDLgNb0EZuYbSywsM3+66fPqsLcHZ+ZXImIRvd33vxIRJwKf5J4T5lt7yOyXD9G5PS6i86rlGpw29I6d0yN7x85pEDunjOKdA+7rFGbvNEMbOgdGtHfsnKKKds6oPmXpgZl59ba2DVpEPBf4G+CbmfmKiNgHeFtmHj3gpfVVRHydLRwKl92/kNXLgOOBHTNz34h4MPDBzPytLvO2dD/JJv5PTEScn5mPGvQ61I7esXPuqdvOqTJHsnfsnOawc5rNfZ1y7J1maEPnwOj2jp1TTunOGdWBzMWZechm2y7KzEMHtabNRefVsE/IzH8a9FruS0R8BHhVZt5end4BeEf28IrqETH7dpgCjgY2ZuafdZl3KXAYcH5mHlxtuyIzf6PbNbZFRLwVGAdOB9Zt2p6ZFw9sUSOq6b1j5/xKz51TZV7KCPaOndMcdk4Z/eicKsd9nULsnWZoeufAaPeOnVNO6c4ZqacsRcR+dN4+bHlEPGfWl5Yx6y3FmiAzpyPi94BGFwZwwKayAMjM2yLi4F4CM/OizTadFxHf6SFyXWau77w2FETEAu7jxai2JSL+z5a2Z+ZHu83so03T25WztiXQ9f/8a37a0jt2zj302jkwur1j5wyYnVNc8c6pctzXKcfeGaC2dA6Mdu/YOUUV7ZyRGsgAD6XzQkTbA8+ctX0V8LJBLGgbzouI93Hv59I1aeI/FhE7ZOZtABGxIz3er6qMX+UDhwLLe4j8RkT8JbAoIn4beAXwnz3kzX6F8yngt4CLgcYVRmY+cdBrUKt6x84p0zkwor1j5zSCnVNW8c6ZlfOry8B9na7ZOwPXps6BEe0dO6ec0p0zqk9ZenRmfmvQ69iW6rl+m8teXtegtGqa+ZfAp4AAngu8JTM/1kPm1XSmjEHn1bWvBt6Umd/sMm8M+APgyVXml4GTs9CdPyK2Bz6RmU8pkVdSROwK/D/g/pn51Ih4OPDozPzQgJc2ctrQO3ZOmc6pMkeyd+yc5rBzyuhH51S57usUYu80Qxs6B0a3d+ycckp3zqgOZP4ReDOwBvgScADwmsz894EurKUiYn9g06Twa5n5vUGup24RMQF8NzMfOui1bC4i/gv4MPD6zDywOpzwklF4fmfT2DvljHrnQHN7x85pDjunHDunuZ0D9k5T2DlljXrvjFLnjNpTljZ5cmb+WUQcBVwDPAc4B2hcYUTnvef3Z9ZzMDPzTV1mraBz6ODe3PM903t6YbrMvDIibtq0xojYMzOv7TavegD+IfD4atPZwL9k5oYu8zZNhO8hu3zV7oj4z1l548DDgNO6yarBzpl5WkS8DiAzN0aEbwk5GK3oHTsH6LFzqsxR7R07pzlGrnOqvOK9U7pzqgz3dcqxd5qhFZ0D7utUm87GzulW0c4Z1YHMRPXv04FPZeYdm16QqEki4oPAYjrT0ZPpHK7Wy4svfQ44F/gKBd4zHSAingW8A7g/cCOwF3AVnZLr1gfo3Ebvr06/uNr20i7zZr/g0hTwPGDHrZx3Lt4+6/ONwE8z8+c95PXT6ojYiargIuJw4I7BLmlkNb537JxinQOj2zt2TnOMYudA4d7pU+eA+zol2TvN0PjOAfd1sHNKKNs5mTlyH8Bbge8Dl9C5Y66g85ZdvWQeARxXfb4CeGCBdV6+2b9LgHN7yLu0Dz/Ly4Cd6BymBZ1y+1CvmXPZ1uNlXNTj9+9K5wXMngHsUvrnWvB6HgKcV5XEecAP6bxy+8DXNmofpXvHzmlX51SZQ987dk5zPtqwr1O6c6qMSwv/HIt3zqbcuWzr8TKGvnOqddo7DfhoQ+dUOe7r3Me2Hi/DzuniYySPkMnMv6ie53hHdt7+7G7gd7vNi4g30JkSPpTO88km6Bye99gel7qm+vfuiLg/cAuwWw95X4iIp2XmF3tc12wbMvOWiBiLiLHM/HpEvKvHzOmI2DczfwwQEfvQw8Q5Ig6ZdXKMzm3V9X0/Ip4PvI3OoX4BvDci/jQzP91tZr9k5sUR8Zt07psB/CB7eBqGuleyd+ycZndOlTGSvWPnNEdL9nVKdw6U751+dA64r1OMvdMMLekccF/HzulR6c4ZuYFMRCwGHpyZl83avBO97XwfBRxM5625yMxfRMTSHvI2+UJ0XmH6H4FN7x1/cg95rwJeFxHrgQ107kCZmct6yLw9IpbQOVTv1Ii4kVlvIdelPwW+HhE/qU7vDRzXQ947+PVzEjfSeV7r83rIez3wyMy8EX713NGvAI0qjM3u61dW2/aMiOnMvG6wqxstfegdO6fZnQMj2Dt2TnO0aF+ndOdA+d7pR+eA+zpF2DvN0KLOAfd17Jwe9KNzRm4gQ+eBcnpEHJCZm+7YJ9N5a7Fui3t9ZmZEbHoe2XYF1gmd59L9IfA44Ft0HpQf6CFvOXAMncP93hQRe9L7/0R9vcp9FfCi6vOuX4yvch7wL3Tef/52Om+j1svb6H2BX7/NG9Xnz9j0vNbMfOc888Y2lUXlFjqT4abpx31d3Sl9W9g5ze4cGM3esXOaoy37OqU7B8r3Tj86B9zXKcXeaYa2dA64r2Pn9Kb4fb2JV7KvqsOJzgCeD52JFrAiMy/sIfa0iPgXYPuIeBnwVXr/Hx6Aj9B58ab3AO8FHg58tIe8fwYOB36vOr0KeF8vC6Qz1DuLzuFlS4FPZuYtPWZ+FHgg8Hd0rvc+wMd6yDuUTvHuRufFsV5O57l/S6uP+fpSRHw5Io6NiGOBLwL/1cP6+qJP93V1oQ+3hZ3T7M6BEewdO6c5WrSvU7pzoHzv9KNzwH2dIuydZmhR54D7OnZOD/pxX4/qhWlGSkTsB5yUmY+PiL8C7szM9/SY+dvAk6uTX87MrxRY5/cy8+Hb2jaPvIsz85CIuCQzD662XZaZBxZY6wHAC4CjgZ9n5pE9ZJW+3ucAT8/MVdXppcCZmfn4+/7O+8x8Dr9+Duu5mfnZbrP6qR/3dXWn9G1h5zS3c6rvH8nesXOaow37On167PWld0p2TpXnvk4h9k4ztKFzqkz3de5j2zzy7JxC9/VRfMoSmfn96HgI8EI6h6zNW0R8MzOPiIhV3POQrZdHxAxwK/C2zHz/VkPu28URcXhmfru6vEcBvUyaN0TEOL9+i64VwEwPebPdCNxA5/CyXXrMKn29dwXWzzq9vto2L/dxex9f6PYurtR9Xb0rcVvYOffQ5M6BEe0dO6c5WrKv04/HXr96p2TngPs6xdg7zdCSzgH3deycHpXunJE8QgYgOodCvQS4LjN/bxtn7/YydgL+JzMf2uX3X0Xn1ZuvrTbtCfyAzgsnZWYeMM+8Y+hMWQ+hc7jec4G/ysxPdbO+KvMVdA7ZWgF8CjgtM7/XbV6VWfp6v75a4xnVpmfTOfTv73tZ5xYup6fbeyuZ98vMG3rMOJY+39c1N/2+LeycrjOLXu8qc2R7x85pjqbv6/TpsVe0d/rROVWu+zq/znRfZ0g0vXOq73dfp8PO6S3jWArd10d5ILMYuB44usThb/dxObtl5vVdfu9e9/X1zPxpF5n70XkxpwC+mplXdbO2WXl/T+fBd2kvOZtl9uN6H8Kvp5fnZOYl3axtDpfT9e29lbwzM/PpPWbUcl/XttVxW9g5XWUWv95V7kj2jp3THE3f1+njY69Y7/Sjc6pc93V+nee+zpBoeudU3+u+zhbYOfPOKHZfH9mBjCRJkiRJ0qCM3LssSZIkSZIkDdrID2Qi4ngzm5k3ypltWKO6N6r3lzZktmGNbcm0c5pjVO8vo5rZhjW2KVPz15bb1sdeczPbsMZSmSM/kAH6UdyjmtmGNbYlsw1rVPdG9f7Shsw2rLEtmXZOc4zq/WVUM9uwxjZlav7actv62GtuZhvWWCTTgYwkSZIkSVLNhvZFfSfHF+WiBcu2eb7102uYHF80p8x1uy+Y0/mm71zN+LLt5nTeiZtj22cCNqxfzcTktjPH1qzf5nk2WT+zhsmxOVz3mbndR9bnWiZjam4XPsf73bwyF07OLXPjaiYXzO32YXp6bplzvB/lho1zytuQa5mY4/WOuV7v6buZHF+8zfOt2XAH6zfePbc7pn5lcmxunQNzf+ztud+tc8q79dYZdtxxbvP1a6/aYU7nm+sac3pu92mADbmOiVi4zfPFxMScMwf62Jss+9ibjzlnzuP2WT+zlsmxbV/3nJrj9d6wmsmJuXXtqrt+cXNmrpjTmfUrkzGVU7Htn/F87tcP+Y3VczrfTbdMs2Kn8Tmd94eXb/u+uoF1TLDtfpiPUc1swxrnmxljc/sdN9f9xjUzd7E+17qvM08LprbLyaU7bvN8G9euZsHU3Pp/513vmNP57rptPUt2mNvvn9uvnNvfbHO9D5a+/wHkzMyczjfox96gMufVDwvm9rtorvs5ABuXzfF861azYOG27+vrVt/KxrWrt9g5c7u3ttCiBct4zP1+v2jmj/9hbn/IzMf9Ty77YFh0xc+L5gHk2nXlM9eVz+RBexaPHLtjbjumczV9w41F8wDGHrR30bxv/ehDRfNGxaIFy3jMri8smvmeM08rmgfwysOeUzRv5pa5DY3mY/x+9yueOX3jTcUzx/bavXhmcbfeXjxyw8PKd+3Xznl9V29vPOqmYjsOn3hK0cwvffk7RfMAfuf+B5UNjD78Hd2P/6DsxzqjDwe359z+MByksUVz+8/Tufr2mjOL5o2KyaU7st9Rrymaedxrv1A0D+DzD9+paN7YorL/qQIws2ZN8cy+9Fg/FO7G8e3L/41+61MeUjTvyi++a6tf8ylLkiRJkiRJNXMgI0mSJEmSVDMHMpIkSZIkSTVzICNJkiRJklQzBzKSJEmSJEk168tAJiKeHREP7+L7nhARj+nHmiQNLztHUt3sHUl1snOk4dTVQCYiJiPivt5w+9nAvAojIhYATwAeM2tb+fewktQ6do6kutk7kupk50ijaV4DmYh4WES8A/gB8JBq21sj4nsRcXlEvL2awD4LeFtEXBoR+0bEyyLigoi4LCI+ExGLq+89JSI+GBHnA6cBLwdeU33f44AXRMR3I+JPImJFySsuqfnsHEl1s3ck1cnOkUbbgm2doZrUPh/4g2rTh4G/zcxVEbETcBSwX2ZmRGyfmbdHxOeBL2Tmp6uM2zPzX6vP31xlvbfK2x14TGZOR8TfAndl5turr50bEWcCxwLnRMSVwMnAWZk50/O1l9Q4do6kutk7kupk50jaZJsDGeB64HLgpZn5/c2+dgewFvhQRHwB+MJWMh5RFcX2wBLgy7O+9qnMnN7ahWfmz4C/q77/qcC/ARfSmRLfQ0QcDxwPMDW+dNvXTFIT2TmS6tbO3mHxtq+ZpCZqZedMLPHZTlJpc3nK0nOB64DTI+JvImKvTV/IzI3AYcCngWcAX9pKxinAH2fmbwBvBKZmfW31thYQEYcB7wfeQ+fQu9dt6XyZeVJmrszMlZPji7YVK6mZ2tk5Y3aO1GKt7J2JmNrSWSQ1Xys7Z8HUfb3EjaRubPMImcw8CzirOnzuRcDnIuJm4KXAzcDizPxiRJwH/KT6tlXA7P8uXgpcHxETwDF0CmhLVgHLNp2IiCcDbwduoHMo3asyc/08rp+klrFzJNXN3pFUJztH0iZzecoSAJl5C/Bu4N3VRHWaThF8LiKmgABeW539E8C/RsQJdCbAfw2cD9xU/bu1Y/v/E/h0RPwu8ErgFuCZmfnT+V4xSe1m50iqm70jqU52jqQ5D2Rmy8zvzDp52Ba+fh73fFu2D1Qfm5/v2M1O/xA4oJs1SRpedo6kutk7kupk50ijaV5vey1JkiRJkqTeOZCRJEmSJEmqmQMZSZIkSZKkmjmQkSRJkiRJqpkDGUmSJEmSpJp19S5LbZATC9j4gB2LZm731e2K5gGMrV9XNC83ThfNA2Byonhk3G/n4pkbdlhUPHNi3caieTE+XjQPYGZR2dsnx6Jo3siYmSHvXlM08hV7P65oHsBP3rpv0byHvG+yaB7A9M7Li2eu/Y3dimf+4ojyv0L3/Y/biubF9EzRPIAFl/6oeKa6EwvGGd+57L7OUx/82KJ5ADOPe0jRvIkbVxXNA2C6/P7T+j12KJ45M17+d/TUZdeWDczyvZNr1hbP1PxNT8FtD8+imV982sFF8wAW7F748byx7N8DAGM7le1ugJmdlhXPZGP5x/PYzWX3ddYcsEfRPIBlV5fdpx9ft/Wfo0fISJIkSZIk1cyBjCRJkiRJUs0cyEiSJEmSJNXMgYwkSZIkSVLNHMhIkiRJkiTVzIGMJEmSJElSzVo5kImI/xn0GiSNDjtHUt3sHUl1snOkwWjlQCYzHzPoNUgaHXaOpLrZO5LqZOdIg9HKgUxE3DXoNUgaHXaOpLrZO5LqZOdIg9HKgczWRMTxEXFhRFy4YcPqQS9H0pCb3Tnrc+2glyNpBNyjd2bWDHo5kobc7M6ZXu3fV1JpQzWQycyTMnNlZq6cmNhu0MuRNORmd85kTA16OZJGwD16Z2zRoJcjacjN7pzx7fz7SiptqAYykiRJkiRJbeBARpIkSZIkqWYOZCRJkiRJkmrWyoFMZi4Z9BokjQ47R1Ld7B1JdbJzpMFo5UBGkiRJkiSpzRzISJIkSZIk1cyBjCRJkiRJUs0cyEiSJEmSJNXMgYwkSZIkSVLNFgx6Af0Sa9YxdvmPimau+t0Di+YBTN02WTRv4oK7i+YBjG2/vHhmjpWfBa7dsezPEmDi+iyaN3N3+dtn/Gc3Fs2L9RuL5o2KnJ5m+vbbC4eWvf8B/PXvfqpo3ifftbJoHsDYNb8on7nTPsUzF98QxTPHVq0umrex9H0SiPHx4pnqTm7YyMbrbxj0MrbpRy8p+zv/ocdfWzQPgAMeXDxy4qbyv/Only0snpmry/ZOP/Z1SsucGfQSWmnhdXfz4D+/uGjmxg3ri+YB3Px/H100735f+nnRPACi/D5EP/Ybj/rUOcUzTz9g96J5E/9d9m+hvphZs9UveYSMJEmSJElSzRzISJIkSZIk1cyBjCRJkiRJUs0cyEiSJEmSJNXMgYwkSZIkSVLNHMhIkiRJkiTVzIGMJEmSJElSzRzISJIkSZIk1ayxA5mI2Dsivh8Rp0TEDyPi1Ig4MiLOi4j/jYjDBr1GScPDzpFUN3tHUp3sHKl5GjuQqTwIeAewX/Xx+8ARwInAX25+5og4PiIujIgL1+faWhcqaSh03TkbWFfrQiUNDXtHUp267xz/vpKKa/pA5urMvCIzZ4Arga9mZgJXAHtvfubMPCkzV2bmysmYqnmpkoZA150zwcKalyppSNg7kurUfef495VUXNMHMrP/62dm1ukZYEH9y5E05OwcSXWzdyTVyc6RGqTpAxlJkiRJkqSh40BGkiRJkiSpZo09LC0zrwEeMev0sVv7miT1ys6RVDd7R1Kd7BypeTxCRpIkSZIkqWYOZCRJkiRJkmrmQEaSJEmSJKlmDmQkSZIkSZJq1tgX9e1Vzswws3Zd0cwHvv47RfMAxvfdq2hebL+8aB7A7Y/Zo3jmDn90bfHMpS+4rXjmzO4riubFgvIPuZnb7yial9PTRfNGRUQwtnBh0cyZdWU7DOAfPvr8onm77l9+jRN3bSie+fPfmiyeue+ptxbPzKmy96EF99+taB5Ari9/+3Bj+Ug1R6xu/u7mzYcsK5654mOXFM+cWLFz8UyWLS2bNzNTNg+YWbu2eKa6kZCFb9+IsnnA/b58XdG8vOPOonkAM/vuXjzzS/95avHMpz/mWcUzx/coe0zIzE23FM0DyPXrywau2/r93CNkJEmSJEmSauZARpIkSZIkqWYOZCRJkiRJkmrmQEaSJEmSJKlmDmQkSZIkSZJq5kBGkiRJkiSpZgMdyETEmyLiyEGuQdJosXck1cnOkVQnO0dqlwWDvPDM/JtBXr6k0WPvSKqTnSOpTnaO1C7Fj5CJiL+OiB9ExDcj4uMRcWJEHBQR346IyyPijIjYoTrvKRHx3OrzayLijRFxcURcERH7VdtXRMR/R8SVEXFyRPw0InYuvW5J7WXvSKqTnSOpTnaONLyKDmQi4pHA0cCBwFOBldWXPgr8eWYeAFwBvGErETdn5iHAB4ATq21vAL6WmfsDnwb2LLlmSe1m70iqk50jqU52jjTcSh8h81jgc5m5NjNXAf8JbAdsn5nfqM7zEeDxW/n+06t/LwL2rj4/AvgEQGZ+CbhtaxceEcdHxIURceEG1vV0RSS1xsB6Z3bnrLdzpFHhvo6kOjWnc9LOkUpr2rssbXqUT9PF69tk5kmZuTIzV06wsOzKJA2rrntndudM2jmS5sZ9HUl1Ktc5YedIpZUeyJwHPDMipiJiCfAMYDVwW0Q8rjrPi4FvbC1gK5nPB4iIJwM7FFyvpPazdyTVyc6RVCc7RxpiRd9lKTMviIjPA5cDv6TzfMY7gP8P+GBELAZ+Ahw3j9g3Ah+PiBcD3wJuAFaVXLek9rJ3JNXJzpFUJztHGm79eNvrt2fm31blcA5wUWZeChy++Rkz89hZn+896/MLgSdUJ+8AficzN0bEo4FHZvoERkn3YO9IqpOdI6lOdo40pPoxkDkpIh4OTAEfycyLe8zbEzgtIsaA9cDLel2gpKFj70iqk50jqU52jjSkig9kMvP3C+f9L3BwyUxJw8XekVQnO0dSnewcaXg17V2WJEmSJEmShp4DGUmSJEmSpJo5kJEkSZIkSapZP17UtzlmpovGxcKFRfMAvv/KFUXzHvpnlxXNA9j+f4pHsvHqnYpnXn3C0uKZe//n6rKBGzeWzQOYLns/J7Ns3ojITGbWri2aGQvKV/TG7crevlMX/aRoHkCuX188c9kjDiieuf5ddxfPnDq27ON543W/KJoHwNh4+Uw1R0TxyL0/W/h3X86UzQOi8K9SgNhucfHMWx6/e/HMZT9ZUzQvzr+5aJ4aJCEL73fGgomieQC3P3K3onnLr5wqmgewdtfy/XDQW19RPHPZgeX/dpleWPaYkGWfv6FoHvRhX/Q+/r7yCBlJkiRJkqSaOZCRJEmSJEmqmQMZSZIkSZKkmjmQkSRJkiRJqpkDGUmSJEmSpJo5kJEkSZIkSaqZAxlJkiRJkqSaOZCRJEmSJEmqWeMGMhHx2oj4bvXx6ojYOyKuioh/jYgrI+KsiFg06HVKGg52jqS62TuS6mTnSM3VqIFMRBwKHAc8CjgceBmwA/Bg4J8zc3/gduDoQa1R0vCwcyTVzd6RVCc7R2q2BYNewGaOAM7IzNUAEXE68Djg6sy8tDrPRcDeW/rmiDgeOB5gisX9Xquk9rNzJNXN3pFUJztHarBGHSFzH9bN+nyarQySMvOkzFyZmSsnWFjPyiQNIztHUt3sHUl1snOkBmjaQOZc4NkRsTgitgOOqrZJUj/YOZLqZu9IqpOdIzVYo56ylJkXR8QpwHeqTScDtw1uRZKGmZ0jqW72jqQ62TlSszVqIAOQme8E3rnZ5kfM+vrb612RpGFm50iqm70jqU52jtRcTXvKkiRJkiRJ0tBzICNJkiRJklQzBzKSJEmSJEk1cyAjSZIkSZJUMwcykiRJkiRJNWvcuywVExALmn/19nvL1UXzsmhax09eunfxzAV3F49k50uni2fOLCg7sxzvw30yp8tfb3WhD50zvusuRfMA9j31lqJ5Nz5nv6J5ADtdubp45i2P2lg8c8k7di2eObPjqqJ5Y3csLZoHkGvXFc/EGuteRNm8LL8nsWFp2W6cjPL/n3jzyvJ3whWnl8/c8eKyHQ4Qd60pmle+bdUYEcTkZNHIXFf+d8qG7cp2RP7s+qJ5ALHn8uKZS35RvnNueXj5v122/1HZdeb0TNG8unmEjCRJkiRJUs0cyEiSJEmSJNXMgYwkSZIkSVLNHMhIkiRJkiTVzIGMJEmSJElSzRzISJIkSZIk1cyBjCRJkiRJUs0cyEiSJEmSJNWsNQOZiPhsRFwUEVdGxPGDXo+k4WfvSKqTnSOpTnaONHgLBr2AeXhJZt4aEYuACyLiM5l5y+wzVEVyPMAUiwexRknD5T57x86RVJj7OpLqZOdIA9aaI2SAEyLiMuDbwB7Agzc/Q2aelJkrM3PlRCysfYGShs599o6dI6mw+e3rYO9I6sk8/76aqn2B0rBrxREyEfEE4Ejg0Zl5d0ScDdgIkvrG3pFUJztHUp3sHKkZ2nKEzHLgtqos9gMOH/SCJA09e0dSnewcSXWyc6QGaMtA5kvAgoi4CngrncPqJKmf7B1JdbJzJNXJzpEaoBVPWcrMdcBTB70OSaPD3pFUJztHUp3sHKkZ2nKEjCRJkiRJ0tBwICNJkiRJklQzBzKSJEmSJEk1cyAjSZIkSZJUMwcykiRJkiRJNWvFuyx1JSE3biybWToPiIiieTk+XjQP4P7fXFc88z0fel/xzBP3P7J4Jg/as2jczEwWzVOD9KFzNl73i6J5ADMPOrho3s6X31U0D2Bmovz/FfzDb55WPPOUvzyseOb0vrsVzYu15fs7Jvqw67C+fKSaY2xj4d99OVM2D4jpsvtj/RJryj+mZ3ZYUjQvbizfEbluunimGmKs/N8uO116R9G8se2XF80DGO/DfXqn119dPHPRX+5RPHPi+tvLBvbh9pm5vex9iA1b/x3jETKSJEmSJEk1cyAjSZIkSZJUMwcykiRJkiRJNXMgI0mSJEmSVDMHMpIkSZIkSTVzICNJkiRJklSzxg5kIuKaiNh50OuQNDrsHUl1snMk1cnOkZqnsQMZSZIkSZKkYdXTQCYi/k9EXB4Rl0XExyJi74j4WrXtqxGxZ3W+UyLiubO+767q3ydExDkRcWZE/CAiPhgR91pTRLwoIr4TEZdGxL9ExHgv65bUXvaOpDrZOZLqZOdIo6XrgUxE7A/8FfCkzDwQeBXwXuAjmXkAcCrwnjlEHQa8Eng4sC/wnM0u52HAC4DHZuZBwDRwzFbWdHxEXBgRF25gXVfXS1JzNa137BxpuDWtc6rz2jvSkGp85+Tarq6XpK3r5QiZJwGfysybATLzVuDRwH9UX/8YcMQccr6TmT/JzGng41v4nt8CDgUuiIhLq9P7bCkoM0/KzJWZuXKChfO9PpKar1G9Y+dIQ69RnVOtwd6RhlezOyem5nt9JG3DgpouZyPV8Kc6ZG5y1tdys/NufjroTIVf17/lSRpC9o6kOtk5kupk50hDoJcjZL4GPC8idgKIiB2B/wFeWH39GODc6vNr6ExhAZ4FTMzKOSwiHlgVyQuAb252OV8FnhsRu2y6nIjYq4d1S2ove0dSnewcSXWyc6QR0/URMpl5ZUS8BfhGREwDl9B5ruKHI+JPgZuA46qz/yvwuYi4DPgSsHpW1AXA+4AHAV8Hztjscr4XEX8FnFWVygbgj4Cfdrt2Se1k70iqk50jqU52jjR6enrKUmZ+BPjIZpuftIXz/RI4fNamP5/1+Z2Z+YwtfM/esz7/JPDJXtYqaTjYO5LqZOdIqpOdI42Wnt72WpIkSZIkSfNX14v6blFmng2cPcg1SBot9o6kOtk5kupk50jt4hEykiRJkiRJNXMgI0mSJEmSVDMHMpIkSZIkSTUb6GvItE0sKP/jyqXblQ28/Y6yecDCi35UPPPVL/rD4pl/cfnHime+7fl7lw2cmS6bBxBRPlPNMDZePHL1/RYWzdv+rGuK5gFw1+ptn2ee3vRvxxTPPOGczxbP/OwTyj6epzesL5oHkH3IVA+i7P+txYLyvTNx18bimaXte1r5+3UsWVI8c8MDdiyeedNBi4vm7XbNL4rmAUyvW1c8U13IJNeXfazE5GTRPICxG24pmjezyw5F8wDG1pb/m+DaO8uvc+dbyu+T3XDk/Yrm7XLyBUXzAHImCwduPc8jZCRJkiRJkmrmQEaSJEmSJKlmDmQkSZIkSZJq5kBGkiRJkiSpZg5kJEmSJEmSata6gUxEnBARV0XEqYNei6ThZ+dIqpOdI6lu9o40OG182+tXAEdm5s8HvRBJI8HOkVQnO0dS3ewdaUAafYRMRLw2Ir5bfbw6Ij4I7AP8V0S8ZtDrkzRc7BxJdbJzJNXN3pGapbFHyETEocBxwKOAAM4HXgQ8BXhiZt48wOVJGjJ2jqQ62TmS6mbvSM3T2IEMcARwRmauBoiI04HH3dc3RMTxwPEAUyzu+wIlDRU7R1Kd5t051fnsHUndcl9HaphGP2VpvjLzpMxcmZkrJ1g46OVIGnJ2jqS62TuS6mTnSP3V5IHMucCzI2JxRGwHHFVtk6R+sHMk1cnOkVQ3e0dqmMY+ZSkzL46IU4DvVJtOzsxLImKAq5I0rOwcSXWycyTVzd6RmqexAxmAzHwn8M7Ntu09mNVIGnZ2jqQ62TmS6mbvSM3S5KcsSZIkSZIkDSUHMpIkSZIkSTVzICNJkiRJklQzBzKSJEmSJEk1cyAjSZIkSZJUs0a/y1LT5MaN5UNvuqVoXE7PFM0DyAftUTzzv087pXjm0w74reKZsXsWzct+vK1gll2jelD69s3yj+dlP1xVNC/32q1oHkD04T793RPeXzzz6Y95VvHMmCx7m8fChUXzAGJ8vHgmq8tHjoyZ6aJxWTgPYNUeZe+H20+XX+NPji7/WHnIX95WPHPBXXcVz9x17e5F87IPt48apPDv6Fy3rmgewOpD9iyaN/XflxXNAxh7xIOLZz57j8uLZ35l98cVz7zfOWX//p2e6cPfQn34Xbg1HiEjSZIkSZJUMwcykiRJkiRJNXMgI0mSJEmSVDMHMpIkSZIkSTVzICNJkiRJklQzBzKSJEmSJEk1q30gExF/GxEn1n25kkaTnSOpTnaOpLrZO1J7eYSMJEmSJElSzWoZyETE6yPihxHxTeCh1baXRcQFEXFZRHwmIhZHxNKIuDoiJqrzLNt0OiJOiIjvRcTlEfGJOtYtqZ3sHEl1snMk1c3ekYZD3wcyEXEo8ELgIOBpwCOrL52emY/MzAOBq4A/yMxVwNnA06vzvLA63wbgL4CDM/MA4OX9XrekdrJzJNXJzpFUN3tHGh51HCHzOOCMzLw7M+8EPl9tf0REnBsRVwDHAPtX208Gjqs+Pw74cPX55cCpEfEiYOOWLigijo+ICyPiwg2s68d1kdR8do6kOtXWOWDvSALc15GGxiBfQ+YU4I8z8zeANwJTAJl5HrB3RDwBGM/M71bnfzrwz8AhwAURsWDzwMw8KTNXZubKCRb2/xpIapNTsHMk1ecUCndO9f32jqStOQX3daRWqWMgcw7w7IhYFBFLgWdW25cC11fPZzxms+/5KPAfVNPbiBgD9sjMrwN/DiwHltSwdkntY+dIqpOdI6lu9o40JLb4vy8lZebFEfFJ4DLgRuCC6kt/DZwP3FT9u3TWt50KvBn4eHV6HPj3iFgOBPCezLy932uX1D52jqQ62TmS6mbvSMOj7wMZgMx8C/CWLXzpA1v5liOAT28qhepFp47oz+okDRs7R1Kd7BxJdbN3pOFQy0BmPiLivcBT6bxiuCT1lZ0jqU52jqS62TtSczVuIJOZrxz0GiSNDjtHUp3sHEl1s3ek5hrkuyxJkiRJkiSNJAcykiRJkiRJNXMgI0mSJEmSVLPGvYbMqMn1G4rmjS1bUjQPYCazeOa6LHu9+2ZmpmhcjI8XzQPIjRuLZ6pL0YIZd+ElTk9Nlg0EFvzoF8Uz75hZUzwzJyeKZ1K4byOiaB4Afegx9aD0bdyH3/k7Xnpb0bycLN87M4umi2f247ES2y8vnjkzVbbLFiyaKpoHwOrV5TM1tKZuLPs7PybK/8m8fsfyj5Nv3/bA4pkLb7ireObtB+xYNG/5T64tmgcws67s34Dcx6/WFvz1IEmSJEmSNFwcyEiSJEmSJNXMgYwkSZIkSVLNHMhIkiRJkiTVzIGMJEmSJElSzRzISJIkSZIk1azxA5mIOCEiroqIUyNiYUR8JSIujYgXDHptkoaPnSOpbvaOpDrZOVJzlH9T9fJeARyZmT+PiMMBMvOgwS5J0hCzcyTVzd6RVCc7R2qIRg1kIuK1wEuqkycD+wH7AP8VEf8OvAxYERGXAkdn5o8HslBJQ8HOkVQ3e0dSnewcqdkaM5CJiEOB44BHAQGcD7wIeArwxMy8OSLOB07MzGdsJeN44HiAKRbXsm5J7WTnSKqbvSOpTnaO1HxNeg2ZI4AzMnN1Zt4FnA48bj4BmXlSZq7MzJUTLOzLIiUNDTtHUt3sHUl1snOkhmvSQEaSJEmSJGkkNGkgcy7w7IhYHBHbAUdV2ySpH+wcSXWzdyTVyc6RGq4xryGTmRdHxCnAd6pNJ2fmJRExwFVJGlZ2jqS62TuS6mTnSM3XmIEMQGa+E3jnZtv2nvX52cDZtS5K0tCycyTVzd6RVCc7R2q2Jj1lSZIkSZIkaSQ4kJEkSZIkSaqZAxlJkiRJkqSaOZCRJEmSJEmqmQMZSZIkSZKkmkVmDnoNfRERNwE/ncNZdwZuLnzxo5rZhjW2JXOQa9wrM1cUvuyhN4/OgeG6vwxbZhvW2JbM+eTZO10Ysn2dNqyxLZltWOOgM+2cLgxZ5/Qjsw1rbEtmG9Y4n8ytds7QDmTmKiIuzMyVZjYvb5Qz27BGdW9U7y9tyGzDGtuSaec0x6jeX0Y1sw1rbFOm5q8tt62PveZmtmGNpTJ9ypIkSZIkSVLNHMhIkiRJkiTVzIEMnGTm/ETEXZudPjYi3tdt3mZZZ0fE7MO+Tqq2nxIRV0fEpdXHQT1cTGN+ljXm9StT3RnV+0vpzuk6c1bW1jonIuItEfHDiLgqIk7o4WIa87OsOdPOaY5Rvb+0oXMAToqIc2ft4/wiIj7bw8U05mc5JJmav7bcto3psQH9ffVbEXFx1TvfjIgH9XAxjflZti1z5F9DRvMXEXdl5pJZp48FVmbmHxfIPhs4MTMv3Gz7KcAXMvPTvV6GpHYZUOccBzwRODYzZyJil8y8sdfLk9R8g+iczc7zGeBzmfnRXi9PUjsMaF/nh8DvZuZVEfEK4LDMPLbXy9P8eISMioqIFRHxmYi4oPp4bLX9sIj4VkRcEhH/ExEPrbYviohPVP8DfQawaKBXQFKr9LFz/hB4U2bOADiMkQT938+JiGXAk4DP9vu6SGqHPvZOAsuqz5cDv+j7ldG9LBj0AtRKiyLi0lmndwQ+X33+buCfMvObEbEn8GXgYcD3gcdl5saIOBL4f8DRdP7ouTszHxYRBwAX38flviUi/gb4KvAXmbmu6LWS1FSD6Jx9gRdExFHATcAJmfm/pa+YpEYa1H4OwLOBr2bmncWujaQ2GETvvBT4YkSsAe4EDi99pbRtDmTUjTWZedCmE5sOqatOHgk8PCI2fXlZRCyhM3X9SEQ8mM40dqL6+uOB9wBk5uURcflWLvN1wA3AJJ3n6v058KZC10dSsw2icxYCazNzZUQ8B/g34HHFrpGkJhtE52zye8DJBa6DpHYZRO+8BnhaZp4fEX8KvJPOkEY1ciCj0saAwzNz7eyN1YtSfT0zj4qIvYGz5xOamddXn66LiA8DJxZYq6T260vnAD8HTq8+PwP4cI/rlDQc+tU5RMTOwGHAUQXWKWl4FO+diFgBHJiZ51ebPgl8qcxyNR++hoxKOwt45aYT8et3Q1oOXFd9fuys858D/H513kcAB2wpNCJ2q/4NOofzfrfckiW1WF86h87rNzyx+vw3gR+WWKyk1utX5wA8l84bGKy9j/NIGj396J3bgOUR8ZDq9G8DVxVbsebMgYxKOwFYGRGXR8T3gJdX2/8R+PuIuIR7Hpn1AWBJRFxF5ylIF20l99SIuAK4AtgZeHNfVi+pbfrVOW8Fjq565+/xEF5JHf3qHIAXAh/vw5oltVvx3snMjcDLgM9ExGXAi4E/7eN10Fb4tteSJEmSJEk18wgZSZIkSZKkmjmQkSRJkiRJqpkDGUmSJEmSpJo5kJEkSZIkSaqZAxlJkiRJkqSaOZCRJEmSJEmqmQMZSZIkSZKkmjmQkSRJkiRJqtn/Dy3WFmlvUV7uAAAAAElFTkSuQmCC\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "translate(sentence_pairs[0], plot='decoder_layer4_block2')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input: je recherche un assistant .\n",
      "target: i am looking for an assistant .\n",
      "attention_weights: dict_keys(['decoder_layer1_block1', 'decoder_layer1_block2', 'decoder_layer2_block1', 'decoder_layer2_block2', 'decoder_layer3_block1', 'decoder_layer3_block2', 'decoder_layer4_block1', 'decoder_layer4_block2'])\n",
      "pred: <start> i m looking for a lot .\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": "<Figure size 1152x576 with 8 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABGEAAAI8CAYAAABRULYhAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABYb0lEQVR4nO3deZglBXn3/d+vt1mYhW1AEGVkB3EEHZHVoEGMG4oaMS5vEJQQH6MhUV/1SZRookbik1f00ogbbjFRRCBuQFBEEZUBhFFANCAuLLIOsy/d9/vHqZZmnp6Z7tOn7jpV9f1cV1/Tp85y32e6zm+q76nFESEAAAAAAACUa6DqBgAAAAAAANqAIQwAAAAAAEAChjAAAAAAAAAJGMIAAAAAAAAkYAgDAAAAAACQgCEMAAAAAABAAoYwAAAAAAAACRjCAAAAAAAAJGAI00K297R9XPH9HNvzq+4JQHOROQAykTkAspE7mA6GMC1j+7WSzpP0sWLRHpIuqKwhAI1G5gDIROYAyEbuYLoYwrTP/5J0lKSHJCkifiFpl0o7AtBkZA6ATGQOgGzkDqaFIUz7rI+IDeM3bA9Jigr7AdBsZA6ATGQOgGzkDqaFIUz7fNf22yXNsf1MSV+W9F8V9wSgucgcAJnIHADZyB1MC0OY9nmrpHskLZf0F5K+IenvKu2oBLY/N5VlAEpH5gDI1IrMkcgdoI+0InfInN5xBHtKoXlsXxsRT5pwe1DS8og4qMK2ADQUmQMgG7kDIBOZ0zvsCdMyto+yfantW2zfavs227dW3Vev2H6b7ZWSlth+qPhaKen3ki6suD2gdcgcAJmanjkSuQP0m6bnDpnTe+wJ0zK2b5Z0hqRrJI2OL4+I+yprqgS23xsRb6u6D6DtyBwAmdqSORK5A/SLtuQOmdM7DGFaxvaPIuKpVfeRwfajJe0paWh8WURcUV1HQPuQOWQOkKlNmSORO0A/aFPukDm9wRCmJWyPH7/3UkmDks6XtH78/oi4toq+ymL7fZJeJulGPTyRjog4obqugPYgcySROUCatmWORO4AVWtb7pA5vcMQZgLblvRVSW+LiJuq7qeXbH9nK3dHRDwjrZkEtn8uaUlErN/mg4GKkDnNQeagDsicZiF30O+anDlS+3KHzOmdoW0/pFWOl/QUSa+R9LcV99JTEfH0qntIdqukYU2YRgN9iMxpDjIHdUDmNAu5g37X2MyRWpk7ZE6PcHWkRzpVnZB4vu1GDqhsv8f29hNu72D7HytsqSxrJP3E9sdsnz3+VXVTwGbInOYgc1AHZE6zkDvod43PHKlVuUPm9AiHIxVs7yzpuxHxeNsfkfTtiDiv6r56zfZ1EXHoZssecc33JrD955Mtj4jPZPcCTIbMIXOATGROszJHInfQ39qSOVJ7cofM6Z3GTiS78CpJXyy+/7Skd0tqYlAM2p41fiyf7TmSZlXcU8+1MQxsnyjp0ohYVXUvmBIyp0HIHNQAmdMwbcsdMqd22pI5Uktyp22ZI5WXOxyO9LBT1AkIRcTVknaz/ZhqWyrFFyRdZvtU26dKulRS4z5Qtve1fZ7tG23fOv5VdV9lsb23pC9JemXVvWDKyJwGIXNQA2ROw7Qpd8icWmpL5kgtyZ02ZY5Ubu5wOJKk4hi+kyLiYxOWPVPSvRFxXWWN9VhxhvI9JD1e0nHF4ksj4uLquiqH7e9Leqekf5X0fEmvljQQEe+otLGSTDju9PiIOKzSZrBNZA6ZU3dkTr2QOc3LHKlduUPm1EtbMkdqV+60KXOkcnOHIUzB9lERceW2ltWd7eUR8YSq+yib7Wsi4skT3+/4sqp76zXbg5JulLRU0vmS3hQR11fbFbaFzGkWMofM6XdkTvO0JXfInHpqS+ZI7cmdtmSOVH7ucDjSwz40xWV1d63tp1TdRIL1tgck/cL264vj+eZV3VRJniPphxGxUtKn1DkTPfofmdMsZA76HZnTPG3JHTKnntqSOVJ7cqctmSOVnDut3xPG9hGSjpT01+rsWjVugaQTI+KJVfRVFts3S9pX0q8krZZkSRERS6rsq9eKILxJ0vbqnAhsgaT3R8SPquyrDLYvkPR/IuIK27Ml/UzSgRGxodrOMBkyh8ypOzKnXsicZmaO1J7cIXPqpW2ZI7Und9qSOVL5ucPVkaQRdSZ4Q5LmT1j+kKSXVNJRuZ5VdQNJFhcnAVulzvGKsv2nkhoVEsUxt9tHxBWSFBHrbJ8n6RmSvlVlb9giMqeZyBwyp1+ROc3V+Nwhc2qpbZkjtSd3Gp85Uk7utH5PGOkPx3x9KSJeXHUvGWwfLWnfiPi07UWS5kXEbVX31Uu2r42IJ21rGVAFMofMATKROc3LHIncQf9qW+ZI7cgdMqd32BNGUkSM2t696j4y2H6nOicY2l+dy8YNS/q8pKOq7KtXbD9bnWP4Hm377Al3LZC0qZquymF7q4EXEddm9YLpIXPInDoic+qLzGlO5kjtyR0yp77alDlS83OnLZkj5eUOQ5iH/cT2RZK+rM6xfJKkiDi/upZKcaKkQyVdK0kRcYft+Vt/Sq3cIWmZpBMkXTNh+UpJZ1TSUXk+UPw5W53gv16dY1CXqPN3cERFfWFqyJxmIHPInLogc5qjLblD5tRbWzJHan7utCVzpKTcYQjzsNmS7lPnWK9xoc4lqZpkQ0SE7ZAk29tV3VAvFZcOu972v0fERkmyvYOkx0TEA9V211sR8XRJsn2+pCdFxPLi9sGSzqywNUwNmdMAZA6ZUyNkTkO0JXfInNprS+ZIDc+dtmSOlJc7DGEKEfHqqntI8iXbH5O0ve3XSjpF0scr7qkMl9o+QZ11/BpJv7f9g4ho2rRWkvYfDwhJioif2j6wyoawbWRO45A56GtkTiO1JXfInBpqUeZI7cmdtmSOVHLucGLeQnHpqVMlPV6dya0kKSJOqaypkth+pqTj1dm16uKIuLTilnrO9nURcajt16gzpX2n7Ruadqk4SbL9RXV28/x8segV6pwM7M+q6wrbQuY0C5lD5vQ7Mqd52pI7ZE49tSlzpHbkTlsyRyo/d9gT5mGfk3SzOpcYe5c6f9E3VdpRSYpQaFwwbGbI9m6SXirpf1fdTMleLekvJb2xuH2FpI9W1w6miMxpFjIH/Y7MaZ625A6ZU0+tyRypNbnTlsyRSs4d9oQpTJjs3RARS2wPS/peRBxedW+9ZPtFkv5Z0i7qTGotKSJiQaWN9Zg716z/e0nfj4jX2d5L0lltulQe+huZQ+YAmcicZmWORO6gv7Ulc6T25A6Z0zsMYQq2fxwRh9m+QtLrJN0l6ccRsVfFrfWU7V9Ken5ENHYS3Ta2j1LnRFF7asLebU1bd5uGzEFdkTn1ROagrsicempL5kjkThOVnTscjvSwc4qzPP+dpIskzVNn0tc0dzc5IGy/JSLeb/tD6pyB/REi4g0VtFW2T6pzebhrJI1W3AumjsxpADKHzKkRMqchWpg7ZE49tSVzpIbnTgszRyo5dxjCPOyy4hJbV0jaS5JsP67alnqn2E1OkpbZ/k9JF0haP35/RDTlcnHjAbis0i5yrYiIb1bdBKaNzGkGMgd1QeY0R9tyh8ypp0ZnjtSq3Glb5kgl5w6HIxVsXxsRT9ps2TUR8eSqeuol25/eyt3R1DOVS5LtAXXOZv1Q1b2Uwfb7JA1KOl+PDP5rK2sK20TmkDl1RebUE5nT3MyRmp07ZE49NT1zpHbnTpMzRyo/d1q/J4ztA9S5dNrCCdNMSVqgCZdTq7uIeHXVPWSy/e+STldn97GrJS2w/cGIOKvazkrx1OLPpROWhaRnVNALtoHMaSYyh8zpV2ROc7Uod8icGmlL5kjty50WZY5Ucu60fggjaX9Jz5O0vaTnT1i+UtJrq2ioTLY/I+mNEfFgcXsHSR9o4KT2oIh4yPYrJH1T0lvVOaavcSEREU+vugdMC5lD5tQamVM7ZE4zM0dqSe6QObXTqsyRWpU7rcgcqfzcaf0QJiIulHSh7SMi4qqq+0mwZDwgJCkiHrB9aIX9lGW4uBTeCyV9OCI22m7ksXe2d5X0Hkm7R8SzbR8k6YiI+GTFrWESZA6ZU3dkTr2QOY3NHKkluUPm1EsLM0dqT+60InOk8nNnoBcv0hAn2l5ge9j2Zbbvsf3KqpsqwUAxnZUk2d5RzRzGfUzSryRtJ+kK23tKauQxi5LOlXSxpN2L27dI+uuqmsGUkTnNQuag35E5zdOW3DlXZE4dtSVzpPbkTlsyRyo5dxjCPOz44sRCz1Nn5dpH0psr7agcH5B0le132363pB9Ien+ZBW0fZftS27fYvtX2bbZvLbNmRJwdEY+OiOdEx+2Smro7684R8SVJY5IUEZvEJRzrgMwpCZlTOjKnnsicEpE7pSJz6qktmSOxrdNEpeZOEyd03Rou/nyupC9HxArbVfZTioj4rO1levikQi+KiBtLLlvqddYnY/uNkj6tzvGnn5B0qDrHLV6SUT/Zats7qXOyKNk+XNKKalvCFJA55SFzykXm1BOZUy5ypzxkTj21InMktnXUvMyRSs4d9oR52H/ZvlnSkyVdZnuRpHVlFrR9tO1XF98vsv24MutNsKOk1RHxYUn3JNRdERHfjIjfR8R9418l1zylmL4fL2kHSa+S9L6Sa1blbyRdJGlv21dK+qykv6q2JUwBmVMeMqdcZE49kTnlInfKQ+bUU3rmSK3KHTKnXKXmjiMaeS6drhTH762IiFHb20maHxF3lVTrnepc8mr/iNjP9u7qTImPKqNelXVd8nXWt1DzhohYYvuDki6PiK/avi4imniSLNkeUuds9Jb084jYWHFLmAIyp7SaZE7JyJx6InNKrUvulIjMqafMzCnqtSZ3yJzylZk7HI4kyfZcSftGxPUTFu+kcnftOlGdXbiulaSIuMP2/BLrVVl3/DrrTy7+tHp4nfUtuMb2JZIeJ+ltxXscK7FeJTZbd39WLHus7dGI+F213WFLyBwyp67InHoic1LqkjslIHPqqaLMkdqVO2ROSTJyhyFMx0ZJ59teEhGri2WfkPR2SWUF/IaICBeX9SqmwxmqqHv5JMvK3gXrVEmHqHM86lJJO6tzluumqWLdxcyROeW6fJJlZE5vkDn1ROaU7/JJlpE7M0fm1FNVP7c25c7lkywjc3qj9PWXc8JIKnYt+qqkl0qdSZekRRGxrMSyX7L9MUnb236tpMvU+eGWxrYlfW2zuv8t6eNl1pW0asLXJkl/ImlxyTVPkfQZSd+S9A+SvqDOlLpRKlp3MUNkDplTV2ROPZE5pWeORO6Ugsyppwp/bm3KHTKnJBnrL+eEKdg+QNI5EfE0238n6aGIOLvkms9U58RGknRxRPx3mfWKmsvVOdHQ8erstnZxRFxadt3NephV1D22xBrLJT1F0g8j4pDi5/ueiHhRWTWrUsW6i5kjc/KQOb1F5tQTmZOL3OkdMqeeqvq5tTV3yJzeKnv95XCkQkTc7I79JL1M0jFl1LH9/Yg42vZKdXYZG79W2+m2xyTdL+msiPhIGfXVOVbxwYh4c0mvPxVzJe1Rco11EbHOtmzPKn6++5dcsxJZ6y56i8xJReb0EJlTT2ROOnKnR8icesr8uZE7ksicnip7/WUI80ifVGeXteUR8UAZBSLi6OLPSU/W5M71yH8gqayQeKqkV9i+XdL4MW6KiCUl1Rufmo7vcjUoaZGkd5VVr/Bb29tLukDSpbYfkHR7yTUnZftRUeKZ4Aulr7soBZlTAjKHzMEWkTklaXPukDnYipSfWxtzp82ZI9U/dzgcaQJ3zoR8p6QXZ+y6tpU+douIO0t67T0nWx4RpX2ANqu5SdLdEbGprHqT1P8jSQslfSsiNmTVnVD/6xHx3JJr9MW6i+npl58bmdPz+mQO+lK//NyaljmT1G1V7pA52JJ++rk1LXfanDlF/VrnDkMYAAAAAACABFwdCQAAAAAAIAFDGAAAAAAAgAQMYSZh+zRqUrOOdat6r5iZtqwrfBapif7QpnWlLe+VmuhnfP6pWee6ZdRkCDO5KlYqajarZlV12Tipp7asK3wWqYn+0KZ1pS3vlZroZ3z+qVnnugxhAAAAAAAA6qixV0camr1djMzfsavnblq3WkOzt5v288aGuyonSRpds1qDc6dfU5Jm3b22q+dtiHUa8eyunhtzZ3VXc+NqjQx39z69dn13NcfWaWSgy/c5a6Sr50nSxk1rNDw0t6vnenSsq+dtGF2jkcHp11y7cYU2jK51V0UhSRoZmBNzhhZ09dwNY2s1MjBn2s/b98AVXdWTpHvuG9WinQa7eu4tN3S3Xm/Ueg2ru+zwQPf/Z9Bt1sVYd59DaQbv1d1/DDfGOg13mema093PZSY5t3LNnfdGxKKungxJ0ohnxWxN/9/UmXwW91uypqvnzSRzpGpyp1vUnKIu8m4mObcuVmtDrGNbZwZGPDvmePqZs0HrNdLlurLvE1Z39Typfts6mjv9bUFJ2rhptYaHuvz9al2Xv1/N5PfIKravJHmwu3VhQ6zViKf/s1k7tlIbxibPnKGuOqmBkfk76oATz0itueZR1eT6nv96fXrNTU/aL73m8PX/k15zbN/HpNeUpMEHuv8Hpxs/+O3nUus10ZyhBTpy15el1vz6xd9IrTfuWbsfkl5zYE53G0MzMdblhslMdLuBMGMH7Zte8tJl/3B7etGGma3t9NTB41NrXnzxNan1xlWROxqo4PM4Nppfs4r3qfy8++HGb6XWa6I53k6Hz35Oas1vXvzD1HrjnvXoQ9Nr+uCD82v+rILfr9Z0N8yfqcEFC1PrXfXQhVu8j8ORAAAAAAAAEjCEAQAAAAAASMAQBgAAAAAAIAFDGAAAAAAAgAQMYQAAAAAAABIwhAEAAAAAAEjAEAYAAAAAACABQxgAAAAAAIAEDGEAAAAAAAASlDKEsf1C2wd18bxjbR9ZRk8Amo3cAZCJzAGQicwBmqOrIYztEdvbbeUhL5Q0rZCwPSTpWElHTli2Qzf9AWgecgdAJjIHQCYyB2iPaQ1hbB9o+wOSfi5pv2LZ+2zfaPsG2/9STFpPkHSW7Z/Y3tv2a21fbft621+xPbd47rm2/832jyR9SdLpks4onneMpJNs/9T239pe1Ms3DqAeyB0AmcgcAJnIHKB9hrb1gGIi+1JJpxaLPi3pzIhYaXsnSSdKOiAiwvb2EfGg7YskfS0izite48GI+Hjx/T8Wr/Wh4vX2kHRkRIzaPlPSqoj4l+K+79n+uqSTJV1h+2eSPiHpkogYm6TX0ySdJknD8xjyAnVVl9yZmDmzB+f3/i8CQIq6ZE7x2g/njub29i8CQIraZs5Wd9QBMFXbHMJIulPSDZJeExE3b3bfCknrJH3S9tckfW0Lr3FwEQ7bS5on6eIJ9305Ika3VDwifiPp3cXzny3pU5KWqTMN3vyx50g6R5LmLnpMbPutAehTtcidiZmzcGRXMgeor1pkTvHYP+TOAu9I7gD1VMvMWTiwE5kD9MBUDkd6iaTfSTrf9jts7zl+R0RsknSYpPMkPU/St7bwGudKen1EPEHSP0iaPeG+1dtqwPZhkj4i6Wx1dqt72xT6BlBf5A6ATGQOgExkDtBi29wTJiIukXRJsWvcKyVdaPteSa+RdK+kuRHxDdtXSrq1eNpKSRP3zZ8v6U7bw5JeoU7oTGalpAXjN2wfL+lfJN2lzm5yb4yIDdN4fwBqiNwBkInMAZCJzAHabSqHI0mSIuI+SR+U9MFicjqqzof/QtuzJVnS3xQP/w9JH7f9BnUmvX8v6UeS7in+3NLJE/5L0nm2XyDpryTdJ+n5EXH7dN8YgPojdwBkInMAZCJzgHaa8hBmooj48YSbh01y/5V65CXUPlp8bf64kze7fYukJd30BKDZyB0AmcgcAJnIHKA9pnWJagAAAAAAAHSHIQwAAAAAAEAChjAAAAAAAAAJGMIAAAAAAAAkYAgDAAAAAACQgCEMAAAAAABAAoYwAAAAAAAACRjCAAAAAAAAJBiquoGyeEwaWhepNRef/bPUen8wMpJecvjG29Nrjq5cmV7zkE9U8zO94YjZuQU3bMyt10RDgxrbaUFqyecc8LTUeuPuOf3x6TV3verB9Jp3HbtDes1drlmTXlOSRm77fSV1MTNj28/V2mOfnFrzOY/Pzblx65+zT3rN2Xfmfx4H73kwvea6/R6VXlOSZv/qvtR6/u1war0mGt1hrlY865DUms9+zl6p9caNHT0rvebguk3pNdcdc1B6zdl3rU6vKUlh5xa8ecuZw54wAAAAAAAACRjCAAAAAAAAJGAIAwAAAAAAkIAhDAAAAAAAQAKGMAAAAAAAAAkYwgAAAAAAACRgCAMAAAAAAJCAIQwAAAAAAEAChjAAAAAAAAAJGMIAAAAAAAAkYAgDAAAAAACQoJZDGNs/qLoHAO1B5gDIRu4AyETmAHlqOYSJiCOr7gFAe5A5ALKROwAykTlAnloOYWyvqroHAO1B5gDIRu4AyETmAHlqOYQBAAAAAACom0YNYWyfZnuZ7WWb1q+uuh0ADTcxczZsInMAlI9tHQCZJmbOxnVkDtALjRrCRMQ5EbE0IpYOzdqu6nYANNzEzBkZInMAlI9tHQCZJmbO8GwyB+iFRg1hAAAAAAAA+hVDGAAAAAAAgAS1HMJExLyqewDQHmQOgGzkDoBMZA6Qp5ZDGAAAAAAAgLphCAMAAAAAAJCAIQwAAAAAAEAChjAAAAAAAAAJGMIAAAAAAAAkYAgDAAAAAACQgCEMAAAAAABAAoYwAAAAAAAACRjCAAAAAAAAJBiquoGyDN6/Wgu/8KPUmqMRqfXGjR19SHrNkV/emV7z4t9dl17zWY8+NL2mJCnW5ZaraN1tkli7TmM33Jxb1M6tV3jgyZsqqLp9esWB0fSSGvzhT/OLSto0RgbU0cCDazTnoqtTa466mv+/m/PdG9NrfuXn30mv+aLHHZ1ec+i3v0uvKUnZ/5JEbEiu2DyDK9drh8tvS6255pDHptYbN/uu1ek11+6+XXrN20/I35bc7/Sb0mtK0tCjds0tuGnLKceeMAAAAAAAAAkYwgAAAAAAACRgCAMAAAAAAJCAIQwAAAAAAEAChjAAAAAAAAAJGMIAAAAAAAAkYAgDAAAAAACQgCEMAAAAAABAAoYwAAAAAAAACRjCAAAAAAAAJGAIAwAAAAAAkIAhDAAAAAAAQIK+HcLYXmz7Ztvn2r7F9hdsH2f7Stu/sH1Y1T0CaBZyB0AmMgdAJjIH6A99O4Qp7CPpA5IOKL5eLuloSW+S9PYK+wLQXOQOgExkDoBMZA5QsaGqG9iG2yJiuSTZ/pmkyyIibC+XtHjzB9s+TdJpkjRbczP7BNAcU84dMgdAD7CtAyBT95kzOC+zT6Cx+n1PmPUTvh+bcHtMkwyQIuKciFgaEUuHNSujPwDNM+XcIXMA9ADbOgAydZ05IwNzMvoDGq/fhzAAAAAAAACNwBAGAAAAAAAgQd+eEyYifiXp4Am3T97SfQDQC+QOgExkDoBMZA7QH9gTBgAAAAAAIAFDGAAAAAAAgAQMYQAAAAAAABIwhAEAAAAAAEjAEAYAAAAAACABQxgAAAAAAIAEDGEAAAAAAAASMIQBAAAAAABIwBAGAAAAAAAgwVDVDZTGlkdGUksOPmqX1HrjbnrZrPSa+35+1/SaR7zp9PSaO+56W3pNSYoF81Lr+VffT63XWAODufXGRnPrFbwuf37/qCvuS69581tzP4eStNtn56bXlKTRVasrqYuZ8cCABubmrjNja9ak1ht3/4uWpNd84r/n19xvjzvSa1Yl7nsgtZ5X8X/PMzY2plizNrXknOtuT6037vZT90mvufDWsfSai65yes3BfR6XXlOS4oEVyQVji3eRRgAAAAAAAAkYwgAAAAAAACRgCAMAAAAAAJCAIQwAAAAAAEAChjAAAAAAAAAJGMIAAAAAAAAkYAgDAAAAAACQgCEMAAAAAABAAoYwAAAAAAAACRjCAAAAAAAAJJj2EMb2qm4K2T7W9tcmWX6C7bd285oAmo/MAZCJzAGQjdwB2mWo6gYi4iJJF1XdB4B2IHMAZCJzAGQjd4D+1vXhSO44y/ZPbS+3fdLWlm/23KfYvs723rZPtv3hYvm5ts+2/QPbt9p+SbF8wPZHbN9s+1Lb3xi/D0A7kDkAMpE5ALKRO0A7zGRPmBdJOkTSEyXtLOlq21dIOnILyyVJto+U9CFJL4iIX9s+ZrPX3U3S0ZIOUGeCe15Ra7GkgyTtIukmSZ+aQe8A6ofMAZCJzAGQjdwBWmAmQ5ijJX0xIkYl3W37u5KespXlD0k6UNI5ko6PiDu28LoXRMSYpBtt7zqh1peL5XfZ/s5kT7R9mqTTJGm25s7grQHoQ2QOgEx9lznSZrnj7Wb8JgH0lb7LHTIH6L3sqyPdKWmdpEO38pj1E773dF48Is6JiKURsXTYs7vpD0Cz5GWOZnXTH4BmKTVzpEfmzgjbOgASt3VGBsgcoBdmMoT5nqSTbA/aXiTpaZJ+vJXlkvSgpOdKeq/tY6dR60pJLy6OXdxV0nSeC6AZyBwAmcgcANnIHaAFZnI40lclHSHpekkh6S0RcZftLS0/QJIi4m7bz5P0TdunTLHWVyT9saQbJf1G0rWSVsygdwD1Q+YAyETmAMhG7gAt4IiouocpsT0vIlbZ3kmdye9REXHXlh6/YGCnOHzWs/MalDT4qF1S64276c27p9fc9/Nr02s+tFf+OTd2/PZt6TUlKRbMS6131a8+oxXr7pz2bvFNNu3M8Y7x1MHj8xqUpLHR3HqFX3zoqek1D/joA+k1b35r7udQkg54/S/Ta0rS6KrV6TX/e/Q/r4mIpemF+9R0M0eSFg7uHIfPfV5Og4WxNWtS64178JWHp9e8/wnpJbXfR7d0So/miftyc/2qVRdqxaZ72daZYLq5s3Bo5zhi3gvyGpSkOdUcAnX7qfuk11x461h6zU2z8j8SO1+51X/ayvNA7ozxqgfP14qN90z6FzyTPWGyfc329pJGJL17WxsmADBDZA6ATGQOgGzkDlCB2gxhIuLYqnsA0B5kDoBMZA6AbOQOUI3sqyMBAAAAAAC0EkMYAAAAAACABAxhAAAAAAAAEjCEAQAAAAAASMAQBgAAAAAAIAFDGAAAAAAAgAQMYQAAAAAAABIMVd1AWWzLIyOpNWPd+tR643a7wuk1YzB/frfp5fen1xw7f0V6TUnSg8l1N2zIrddUY6NVd5Bj/qb0kr8/cqf0mjt8P71kdWKs6g7QhRgb09iaNclFI7deYYcvXZte84J/uiy95qn/fEJ6zdH7H0ivWYUYI+dmLKQYzd3Wicfumlpv3O5Xrk2vee/Bc9Jrrt4jP9N3+s8702tKkgaSf38d2/LfLXvCAAAAAAAAJGAIAwAAAAAAkIAhDAAAAAAAQAKGMAAAAAAAAAkYwgAAAAAAACRgCAMAAAAAAJCAIQwAAAAAAEAChjAAAAAAAAAJGMIAAAAAAAAkYAgDAAAAAACQgCEMAAAAAABAgr4Zwth+g+2bbH+h6l4AtAO5AyATmQMgE5kD9KehqhuY4HWSjouI327rgbaHImJTQk8Amo3cAZCJzAGQicwB+lBfDGFs/5ukvSR90/a5ko4pbq+RdFpE3GD7TEl7F8t/LenPqukWQBOQOwAykTkAMpE5QP/qi8ORIuJ0SXdIerqkxZKui4glkt4u6bMTHnqQOtPcSQPC9mm2l9letiHWldw1gDrrRe5MzJyNWp/QNYC6KmNbh9wBsCX8fgX0r74YwmzmaEmfk6SI+LaknWwvKO67KCLWbumJEXFORCyNiKUjnp3QKoCG6Cp3JmbOsGYltQqgAXqyrUPuAJgifr8C+kg/DmG2ZnXVDQBoHXIHQCYyB0AmMgdI1o9DmO9JeoUk2T5W0r0R8VCVDQFoPHIHQCYyB0AmMgfoI31xYt7NnCnpU7ZvUOfEUX9ebTsAWuBMkTsA8pwpMgdAnjNF5gB9o2+GMBGxeMLNF05y/5lZvQBoB3IHQCYyB0AmMgfoT/14OBIAAAAAAEDjMIQBAAAAAABIwBAGAAAAAAAgAUMYAAAAAACABAxhAAAAAAAAEjCEAQAAAAAASMAQBgAAAAAAIAFDGAAAAAAAgARDVTdQlhgb09jKlak1B4eq+etc8PMV6TUHHlyVXvMLT/iP9Jp/te6o9JqSNLj9wtyCG5jH9sTAYNUdpPjxH5+dXvP/+eeT02vGSH6mx+hoek3Ul2eNaGiPPVNrjt1xV2q9cQ+85JD0mqc+JfnfYkl3v2Tv9Jq7XPVAek1JGli5NrWefzuSWq+RIqTkf6cGfn57ar1xt77t8ek19/ncvek1b33ZTuk1/Zjd02tKku5/MLfeem/xLn7zAgAAAAAASMAQBgAAAAAAIAFDGAAAAAAAgAQMYQAAAAAAABIwhAEAAAAAAEjAEAYAAAAAACABQxgAAAAAAIAEDGEAAAAAAAASMIQBAAAAAABIwBAGAAAAAAAgAUMYAAAAAACABAxhAAAAAAAAEjCEAQAAAAAASFCbIYztC2xfY/tntk+ruh8AzUfuAMhE5gDIROYA1RiquoFpOCUi7rc9R9LVtr8SEfdNfEARHqdJ0mzNraJHAM2y1dwhcwD02PS2dYbmV9EjgOaYXuZ4uyp6BBqnNnvCSHqD7esl/VDSYyTtu/kDIuKciFgaEUuHNSu9QQCNs9XcIXMA9Ni0tnVGBhn+ApiR6WUO2zpAT9RiTxjbx0o6TtIREbHG9uWSZlfZE4BmI3cAZCJzAGQic4Dq1GVPmIWSHigC4gBJh1fdEIDGI3cAZCJzAGQic4CK1GUI8y1JQ7ZvkvQ+dXaZA4AykTsAMpE5ADKROUBFanE4UkSsl/TsqvsA0B7kDoBMZA6ATGQOUJ267AkDAAAAAABQawxhAAAAAAAAEjCEAQAAAAAASMAQBgAAAAAAIAFDGAAAAAAAgAQMYQAAAAAAABIwhAEAAAAAAEjAEAYAAAAAACABQxgAAAAAAIAEQ1U3UJaxHbbTquMPT6258LJbUuv9gZ1eMh5YkV7zr598QnpNHbZ7fk1JN508J7XeuvfOSq3XRB4e1tCuu6bW3HTHnan1xj3l62ek1zxwID9zVu21IL2m9j44v6aked+4Pr/o2vySjbNpk+Le+1NLjq1bl1pv3I5fuym95uiD+bmzdpd90mv+/LSF6TUlab+//mVqvdi0IbVeIw0OyPO2Sy05tmJlar1xj7twdXpN35+fObtevX16zZv/9w7pNSVp//9vdm7B1cNbvIs9YQAAAAAAABIwhAEAAAAAAEjAEAYAAAAAACABQxgAAAAAAIAEDGEAAAAAAAASMIQBAAAAAABIwBAGAAAAAAAgAUMYAAAAAACABAxhAAAAAAAAEjCEAQAAAAAASNBXQxjbq7Zx/2LbL8/qB0CzkTkAspE7ADKROUD/6ashzBQslkRIAMiyWGQOgFyLRe4AyLNYZA6Qqi+HMO44y/ZPbS+3fVJx1/skHWP7J7bPqLJHAM1B5gDIRu4AyETmAP1jqOoGtuBFkg6R9ERJO0u62vYVkt4q6U0R8bwKewPQPGQOgGzkDoBMZA7QJ/pyTxhJR0v6YkSMRsTdkr4r6SnbepLt02wvs71s4/rVpTcJoDFmnDkbxtaW3iSARulB7qwrvUkAjUHmAH2iX4cwXYmIcyJiaUQsHZ61XdXtAGi4iZkzMjCn6nYAtMAjc2d21e0AaDgyB+i9fh3CfE/SSbYHbS+S9DRJP5a0UtL8SjsD0ERkDoBs5A6ATGQO0Cf6dQjzVUk3SLpe0rclvSUi7iqWjdq+nhNHAeghMgdANnIHQCYyB+gTfXVi3oiYV/wZkt5cfE28f6OkZ1TQGoAGInMAZCN3AGQic4D+0697wgAAAAAAADQKQxgAAAAAAIAEDGEAAAAAAAASMIQBAAAAAABIwBAGAAAAAAAgAUMYAAAAAACABAxhAAAAAAAAEjCEAQAAAAAASMAQBgAAAAAAIIEjouoeSmH7Hkm3d/n0nSXd28N2qNm+mlXV7bbmnhGxqNfNtAmZ07c1q6pLzW0jd2ZoBrlTt3WlbnWp2Z81yZwZqlnmVFWXms2r2/Pfrxo7hJkJ28siYik1qVm3ulW9V8xMW9YVPovURH9o07rSlvdKTfQzPv/UrHPdMmpyOBIAAAAAAEAChjAAAAAAAAAJGMJM7hxqUrOmdat6r5iZtqwrfBapif7QpnWlLe+VmuhnfP6pWee6Pa/JOWEwZbZXRcS8CbdPlrQ0Il7fg9e+XNKbImLZZstfL+mvJe0taVFEVHECKAAVqSh3viBpqaSNkn4s6S8iYuNM6wHofxVlzifVyRxLukXSyRGxaqb1APS/KjJnwv1nSzplYn3kYE8Y9LsrJR2n7q86AwDT9QVJB0h6gqQ5kl5TbTsAGu6MiHhiRCyR9GtJM/7lCwC2xvZSSTtU3UdbMYRBT9heZPsrtq8uvo4qlh9m+yrb19n+ge39i+VzbP+H7Ztsf1WdX3T+LxFxXUT8Ku+dAKiLEnPnG1FQZ0+YPdLeFIC+VWLmPFQ83sVj2E0dQGmZY3tQ0lmS3pL2ZvAIQ1U3gFqZY/snE27vKOmi4vsPSvrXiPi+7cdKuljSgZJulnRMRGyyfZyk90h6saS/lLQmIg60vUTStVlvAkCtVJY7toclvUrSG3v5hgD0tUoyx/anJT1H0o2S/rbH7wlA/6oic14v6aKIuLMz+0U2hjCYjrURccj4jfFjFoubx0k6aMIHeYHteZIWSvqM7X3V+Z+d4eL+p0k6W5Ii4gbbN5TePYA6qjJ3PiLpioj4Xg/eB4B6qCRzIuLVxf9Of0jSSZI+3as3BKCvpWaO7d0l/amkY3v9RjB1DGHQKwOSDo+IdRMX2v6wpO9ExIm2F0u6vILeADRTablj+52SFkn6ix70CaAZSt3WiYhR2/+hziECDGEAlJE5h0raR9Ivi+HOXNu/jIh9etMypoJzwqBXLpH0V+M3bB9SfLtQ0u+K70+e8PgrJL28eOzBkpaU3iGApikld2y/RtKzJP1ZRIz1tGMAddbzzHHHPuPfSzpBnUMNAKDnmRMRX4+IR0XE4ohYrM7hSwxgkjGEQa+8QdJS2zfYvlHS6cXy90t6r+3r9Mg9rz4qaZ7tmyS9S9I1k72o7TfY/q06J8a8wfYnSnsHAOqmlNyR9G+SdpV0le2f2H5HOe0DqJkyMsfqHFawXNJySbsVjwWAsrZzUDF3Lv4AAAAAAACAMrEnDAAAAAAAQAKGMAAAAAAAAAkYwgAAAAAAACRgCAMAAAAAAJCAIQwAAAAAAEAChjAAAAAAAAAJGMIAAAAAAAAkYAgDAAAAAACQgCEMAAAAAABAAoYwAAAAAAAACRjCtJDtPW0fV3w/x/b8qnsC0FxkDoBMZA6AbOQOpoMhTMvYfq2k8yR9rFi0h6QLKmsIQKOROQAykTkAspE7mC6GMO3zvyQdJekhSYqIX0japdKOADQZmQMgE5kDIBu5g2lhCNM+6yNiw/gN20OSosJ+ADQbmQMgE5kDIBu5g2lhCNM+37X9dklzbD9T0pcl/VfFPQFoLjIHQCYyB0A2cgfTwhCmfd4q6R5JyyX9haRvSPq7Sjsqge3PTWUZgNKROQAytSJzJHIH6COtyB0yp3ccwZ5SaB7b10bEkybcHpS0PCIOqrAtAA1F5gDIRu4AyETm9A57wrSM7aNsX2r7Ftu32r7N9q1V99Urtt9me6WkJbYfKr5WSvq9pAsrbg9oHTIHQKamZ45E7gD9pum5Q+b0HnvCtIztmyWdIekaSaPjyyPivsqaKoHt90bE26ruA2g7MgdAprZkjkTuAP2iLblD5vQOQ5iWsf2jiHhq1X1ksP1oSXtKGhpfFhFXVNcR0D5kDpkDZGpT5kjkDtAP2pQ7ZE5vMIRpCdvjx++9VNKgpPMlrR+/PyKuraKvsth+n6SXSbpRD0+kIyJOqK4roD3IHElkDpCmbZkjkTtA1dqWO2RO7zCEmcC2JX1V0tsi4qaq++kl29/Zyt0REc9IayaB7Z9LWhIR67f5YKAiZE5zkDmoAzKnWcgd9LsmZ47Uvtwhc3pnaNsPaZXjJT1F0msk/W3FvfRURDy96h6S3SppWBOm0UAfInOag8xBHZA5zULuoN81NnOkVuYOmdMjXB3pkU5VJySeb7uRAyrb77G9/YTbO9j+xwpbKssaST+x/THbZ49/Vd0UsBkypznIHNQBmdMs5A76XeMzR2pV7pA5PcLhSAXbO0v6bkQ83vZHJH07Is6ruq9es31dRBy62bJHXPO9CWz/+WTLI+Iz2b0AkyFzyBwgE5nTrMyRyB30t7ZkjtSe3CFzeqexE8kuvErSF4vvPy3p3ZKaGBSDtmeNH8tne46kWRX31HNtDAPbJ0q6NCJWVd0LpoTMaRAyBzVA5jRM23KHzKmdtmSO1JLcaVvmSOXlDocjPewUdQJCEXG1pN1sP6balkrxBUmX2T7V9qmSLpXUuA+U7X1tn2f7Rtu3jn9V3VdZbO8t6UuSXll1L5gyMqdByBzUAJnTMG3KHTKnltqSOVJLcqdNmSOVmzscjiSpOIbvpIj42IRlz5R0b0RcV1ljPVacoXwPSY+XdFyx+NKIuLi6rsph+/uS3inpXyU9X9KrJQ1ExDsqbawkE447PT4iDqu0GWwTmUPm1B2ZUy9kTvMyR2pX7pA59dKWzJHalTttyhyp3NxhCFOwfVREXLmtZXVne3lEPKHqPspm+5qIePLE9zu+rOrees32oKQbJS2VdL6kN0XE9dV2hW0hc5qFzCFz+h2Z0zxtyR0yp57akjlSe3KnLZkjlZ87HI70sA9NcVndXWv7KVU3kWC97QFJv7D9+uJ4vnlVN1WS50j6YUSslPQpdc5Ej/5H5jQLmYN+R+Y0T1tyh8ypp7ZkjtSe3GlL5kgl507r94SxfYSkIyX9tTq7Vo1bIOnEiHhiFX2VxfbNkvaV9CtJqyVZUkTEkir76rUiCG+StL06JwJbIOn9EfGjKvsqg+0LJP2fiLjC9mxJP5N0YERsqLYzTIbMIXPqjsypFzKnmZkjtSd3yJx6aVvmSO3JnbZkjlR+7nB1JGlEnQnekKT5E5Y/JOkllXRUrmdV3UCSxcVJwFapc7yibP+ppEaFRHHM7fYRcYUkRcQ62+dJeoakb1XZG7aIzGkmMofM6VdkTnM1PnfInFpqW+ZI7cmdxmeOlJM7rd8TRvrDMV9fiogXV91LBttHS9o3Ij5te5GkeRFxW9V99ZLtayPiSdtaBlSBzCFzgExkTvMyRyJ30L/aljlSO3KHzOkd9oSRFBGjtnevuo8Mtt+pzgmG9lfnsnHDkj4v6agq++oV289W5xi+R9s+e8JdCyRtqqarctjeauBFxLVZvWB6yBwyp47InPoic5qTOVJ7cofMqa82ZY7U/NxpS+ZIebnDEOZhP7F9kaQvq3MsnyQpIs6vrqVSnCjpUEnXSlJE3GF7/tafUit3SFom6QRJ10xYvlLSGZV0VJ4PFH/OVif4r1fnGNQl6vwdHFFRX5gaMqcZyBwypy7InOZoS+6QOfXWlsyRmp87bckcKSl3GMI8bLak+9Q51mtcqHNJqibZEBFhOyTJ9nZVN9RLxaXDrrf97xGxUZJs7yDpMRHxQLXd9VZEPF2SbJ8v6UkRsby4fbCkMytsDVND5jQAmUPm1AiZ0xBtyR0yp/bakjlSw3OnLZkj5eUOQ5hCRLy66h6SfMn2xyRtb/u1kk6R9PGKeyrDpbZPUGcdv0bS723/ICKaNq2VpP3HA0KSIuKntg+ssiFsG5nTOGQO+hqZ00htyR0yp4ZalDlSe3KnLZkjlZw7nJi3UFx66lRJj1dncitJiohTKmuqJLafKel4dXatujgiLq24pZ6zfV1EHGr7NepMad9p+4amXSpOkmx/UZ3dPD9fLHqFOicD+7PqusK2kDnNQuaQOf2OzGmetuQOmVNPbcocqR2505bMkcrPHfaEedjnJN2sziXG3qXOX/RNlXZUkiIUGhcMmxmyvZukl0r631U3U7JXS/pLSW8sbl8h6aPVtYMpInOahcxBvyNzmqctuUPm1FNrMkdqTe60JXOkknOHPWEKEyZ7N0TEEtvDkr4XEYdX3Vsv2X6RpH+WtIs6k1pLiohYUGljPebONev/XtL3I+J1tveSdFabLpWH/kbmkDlAJjKnWZkjkTvob23JHKk9uUPm9A5DmILtH0fEYbavkPQ6SXdJ+nFE7FVxaz1l+5eSnh8RjZ1Et43to9Q5UdSemrB3W9PW3aYhc1BXZE49kTmoKzKnntqSORK500Rl5w6HIz3snOIsz38n6SJJ89SZ9DXN3U0OCNtviYj32/6QOmdgf4SIeEMFbZXtk+pcHu4aSaMV94KpI3MagMwhc2qEzGmIFuYOmVNPbckcqeG508LMkUrOHYYwD7usuMTWFZL2kiTbj6u2pd4pdpOTpGW2/1PSBZLWj98fEU25XNx4AC6rtItcKyLim1U3gWkjc5qBzEFdkDnN0bbcIXPqqdGZI7Uqd9qWOVLJucPhSAXb10bEkzZbdk1EPLmqnnrJ9qe3cnc09UzlkmR7QJ2zWT9UdS9lsP0+SYOSztcjg//ayprCNpE5ZE5dkTn1ROY0N3OkZucOmVNPTc8cqd250+TMkcrPndbvCWP7AHUunbZwwjRTkhZowuXU6i4iXl11D5ls/7uk09XZfexqSQtsfzAizqq2s1I8tfhz6YRlIekZFfSCbSBzmonMIXP6FZnTXC3KHTKnRtqSOVL7cqdFmSOVnDutH8JI2l/S8yRtL+n5E5avlPTaKhoqk+3PSHpjRDxY3N5B0gcaOKk9KCIesv0KSd+U9FZ1julrXEhExNOr7gHTQuaQObVG5tQOmdPMzJFakjtkTu20KnOkVuVOKzJHKj93Wj+EiYgLJV1o+4iIuKrqfhIsGQ8ISYqIB2wfWmE/ZRkuLoX3QkkfjoiNtht57J3tXSW9R9LuEfFs2wdJOiIiPllxa5gEmUPm1B2ZUy9kTmMzR2pJ7pA59dLCzJHakzutyByp/NwZ6MWLNMSJthfYHrZ9me17bL+y6qZKMFBMZyVJtndUM4dxH5P0K0nbSbrC9p6SGnnMoqRzJV0saffi9i2S/rqqZjBlZE6zkDnod2RO87Qld84VmVNHbckcqT2505bMkUrOHYYwDzu+OLHQ89RZufaR9OZKOyrHByRdZfvdtt8t6QeS3l9mQdtH2b7U9i22b7V9m+1by6wZEWdHxKMj4jnRcbukpu7OunNEfEnSmCRFxCZxCcc6IHNKQuaUjsypJzKnROROqcicempL5khs6zRRqbnTxAldt4aLP58r6csRscJ2lf2UIiI+a3uZHj6p0Isi4saSy5Z6nfXJ2H6jpE+rc/zpJyQdqs5xi5dk1E+22vZO6pwsSrYPl7Si2pYwBWROeciccpE59UTmlIvcKQ+ZU0+tyByJbR01L3OkknOHPWEe9l+2b5b0ZEmX2V4kaV2ZBW0fbfvVxfeLbD+uzHoT7ChpdUR8WNI9CXVXRMQ3I+L3EXHf+FfJNU8ppu/HS9pB0qskva/kmlX5G0kXSdrb9pWSPivpr6ptCVNA5pSHzCkXmVNPZE65yJ3ykDn1lJ45Uqtyh8wpV6m544hGnkunK8XxeysiYtT2dpLmR8RdJdV6pzqXvNo/Ivazvbs6U+KjyqhXZV2XfJ31LdS8ISKW2P6gpMsj4qu2r4uIJp4kS7aH1DkbvSX9PCI2VtwSpoDMKa0mmVMyMqeeyJxS65I7JSJz6ikzc4p6rckdMqd8ZeYOhyNJsj1X0r4Rcf2ExTup3F27TlRnF65rJSki7rA9v8R6VdYdv876k4s/rR5eZ30LrrF9iaTHSXpb8R7HSqxXic3W3Z8Vyx5rezQifldtd9gSMofMqSsyp57InJS65E4JyJx6qihzpHblDplTkozcYQjTsVHS+baXRMTqYtknJL1dUlkBvyEiwsVlvYrpcIYq6l4+ybKyd8E6VdIh6hyPulTSzuqc5bppqlh3MXNkTrkun2QZmdMbZE49kTnlu3ySZeTOzJE59VTVz61NuXP5JMvInN4off3lnDCSil2LvirppVJn0iVpUUQsK7Hsl2x/TNL2tl8r6TJ1frilsW1JX9us7n9L+niZdSWtmvC1SdKfSFpccs1TJH1G0rck/YOkL6gzpW6UitZdzBCZQ+bUFZlTT2RO6ZkjkTulIHPqqcKfW5tyh8wpScb6yzlhCrYPkHRORDzN9t9Jeigizi655jPVObGRJF0cEf9dZr2i5nJ1TjR0vDq7rV0cEZeWXXezHmYVdY8tscZySU+R9MOIOKT4+b4nIl5UVs2qVLHuYubInDxkTm+ROfVE5uQid3qHzKmnqn5ubc0dMqe3yl5/ORypEBE3u2M/SS+TdEwZdWx/PyKOtr1SnV3Gxq/VdrrtMUn3SzorIj5SRn11jlV8MCLeXNLrT8VcSXuUXGNdRKyzLduzip/v/iXXrETWuoveInNSkTk9RObUE5mTjtzpETKnnjJ/buSOJDKnp8pefxnCPNIn1dllbXlEPFBGgYg4uvhz0pM1uXM98h9IKisknirpFbZvlzR+jJsiYklJ9canpuO7XA1KWiTpXWXVK/zW9vaSLpB0qe0HJN1ecs1J2X5UlHgm+ELp6y5KQeaUgMwhc7BFZE5J2pw7ZA62IuXn1sbcaXPmSPXPHQ5HmsCdMyHfKenFGbuubaWP3SLizpJee8/JlkdEaR+gzWpuknR3RGwqq94k9f9I0kJJ34qIDVl1J9T/ekQ8t+QafbHuYnr65edG5vS8PpmDvtQvP7emZc4kdVuVO2QOtqSffm5Ny502Z05Rv9a5wxAGAAAAAAAgAVdHAgAAAAAASMAQZhK2T6MmNetYt6r3iplpy7rCZ5Ga6A9tWlfa8l6piX7G55+ada5bRk2GMJOrYqWiZrNqVlWXjZN6asu6wmeRmugPbVpX2vJeqYl+xuefmnWuyxAGAAAAAACgjhp7Yt6RgTkxZ3DSq5Rt04axtRoZmDPt563fZVZX9SRpdPVqDW63XVfPnXXX2q6etyHWacSzu3ru+kdN/+9Hmtn7HLlzTVfP2xjrNNzl+3z4ym/d1F2vYXe3TnhWd8/bsGmNRobmTvt5azeu0IZNa9xVUUiSRgZmx5yBLjOny8/ivo9f2VU9SbrnvlEt2mmwq+fesry7z/BMPoubdp7+ev2H565draE50+95+L7uslWaQb7O4FO4YWydRga6+/uNWSNdPW/jpjUa7iJzJGnlmjvvjYhFXT0ZkqQRz4rZnv66PZN/n/Z7Qnf/Fs8kcyTplhu6W882ar2G1f32GTVLrOnpB95M/h1ZF6u1IdaxrTMDwyPbxezZO0z7eRs3rtbwcHfbDjvv+WBXz5Oklfdv1Pwdh7t67n0/7fLfxTp+FpNreqD7/UBm8vvr2Lzunrdxw2oNj0x//V239gFt3LB60swZ6qqTGpgzOF9H7PynqTX/53V7p9Ybt9f7f5pe87a/PDi95uJ/uja9pkZH82tKGli8OLXeVb86N7VeE80ZmK8jFp6YWvMbF38ntd64P9nzsPSad5+0NL3mo869Pr2mB7v/JXUmxvZ7bHrNS68+s9RLBrfBbG+nw4eelVrzWxcvS6037lm7H5JfdKCCz+NYNdsdVej2P5y69cP130yt10SzZ++gpU99fWrNUz5yQWq9cZ/d/zH5RVuSOQPzuvtPy5lad9QBqfWuvfLsLd7H4UgAAAAAAAAJGMIAAAAAAAAkYAgDAAAAAACQgCEMAAAAAABAAoYwAAAAAAAACRjCAAAAAAAAJGAIAwAAAAAAkIAhDAAAAAAAQAKGMAAAAAAAAAlKGcLYfqHtg7p43rG2jyyjJwDNRu4AyETmAMhE5gDN0dUQxvaI7e228pAXSppWSNgeknSspCMnLNuhm/4ANA+5AyATmQMgE5kDtMe0hjC2D7T9AUk/l7Rfsex9tm+0fYPtfykmrSdIOsv2T2zvbfu1tq+2fb3tr9ieWzz3XNv/ZvtHkr4k6XRJZxTPO0bSSbZ/avtvbS/q5RsHUA/kDoBMZA6ATGQO0D5D23pAMZF9qaRTi0WflnRmRKy0vZOkEyUdEBFhe/uIeND2RZK+FhHnFa/xYER8vPj+H4vX+lDxentIOjIiRm2fKWlVRPxLcd/3bH9d0smSrrD9M0mfkHRJRIzN+N0D6EvkDoBMZA6ATGQO0G7bHMJIulPSDZJeExE3b3bfCknrJH3S9tckfW0Lr3FwEQ7bS5on6eIJ9305Ika3VDwifiPp3cXzny3pU5KWqTMNfgTbp0k6TZJmD8zb9jsD0K9qkTtkDtAYtcgcabPc0dxtvzMA/aiWmTNr9vbbfGMAtm0qhyO9RNLvJJ1v+x229xy/IyI2STpM0nmSnifpW1t4jXMlvT4iniDpHyTNnnDf6m01YPswSR+RdLY6u9W9bbLHRcQ5EbE0IpaODMzZ1ssC6F+1yJ1HZI5nb343gPqoReYU/fwhd4Y9a1svC6A/1TNzhrd2yhoAU7XNPWEi4hJJlxS7xr1S0oW275X0Gkn3SpobEd+wfaWkW4unrZQ0f8LLzJd0p+1hSa9QJ3Qms1LSgvEbto+X9C+S7lJnN7k3RsSGabw/ADVE7gDIROYAyETmAO02lcORJEkRcZ+kD0r6YDE5HVXnw3+h7dmSLOlviof/h6SP236DOpPev5f0I0n3FH/O1+T+S9J5tl8g6a8k3Sfp+RFx+3TfGID6I3cAZCJzAGQic4B2mvIQZqKI+PGEm4dNcv+VeuQl1D5afG3+uJM3u32LpCXd9ASg2cgdAJnIHACZyBygPaZ1iWoAAAAAAAB0hyEMAAAAAABAAoYwAAAAAAAACRjCAAAAAAAAJGAIAwAAAAAAkIAhDAAAAAAAQAKGMAAAAAAAAAkYwgAAAAAAACRgCAMAAAAAAJBgqOoGyhKzh7Vxv0en1pz3m9RyD9sz931K0q5Xj6bXHFy0c3rNsRUPpdeUpLG5s1LrxYBT6zVRzBrW2N65n8Vn73V4ar1x973q0PSa63dML6lfvmNJes19PvdAek1JGli5rpK6mKF5c7Vpae56+twn7ZRab9yqP12cXnPh9fem19y424L0msO/X5VeU5K8fmNuvd8Op9Zrog07Sbe9Krfm5//0mbkFC4O73p9fdDT/9yuNjqWXXHP4Puk1JUnZv+5spR57wgAAAAAAACRgCAMAAAAAAJCAIQwAAAAAAEAChjAAAAAAAAAJGMIAAAAAAAAkYAgDAAAAAACQgCEMAAAAAABAAoYwAAAAAAAACRjCAAAAAAAAJGAIAwAAAAAAkKCWQxjbP6i6BwDtQeYAyEbuAMhE5gB5ajmEiYgjq+4BQHuQOQCykTsAMpE5QJ5aDmFsr6q6BwDtQeYAyEbuAMhE5gB5ajmEAQAAAAAAqJtGDWFsn2Z7me1lGzaurrodAA03MXM2blpTdTsAWuAR2zob2NYBUK6JmTO6kswBeqFRQ5iIOCcilkbE0pHh7apuB0DDTcyc4aG5VbcDoAUesa0zwrYOgHJNzJzB+WQO0AuNGsIAAAAAAAD0K4YwAAAAAAAACWo5hImIeVX3AKA9yBwA2cgdAJnIHCBPLYcwAAAAAAAAdcMQBgAAAAAAIAFDGAAAAAAAgAQMYQAAAAAAABIwhAEAAAAAAEjAEAYAAAAAACABQxgAAAAAAIAEDGEAAAAAAAASMIQBAAAAAABIwBAGAAAAAAAgwVDVDZRm1VoNfP8nqSV3vXuv1Hrjxm77TXrN717y4/Saf/LYpek1Y9Om9JqSpOtvzq03ui63XgN57Xr5p79MrTm2rpqf2w/e/eH0mi944vHpNW96197pNePm3HXoD2bNqqYuZmblGg1ecX1qyU0xllpv3Jx7H51f9L4H0kvu9fk70mvefuTG9JqSFKOjufXGNqTWa6LZd2zUge/8fWrNTb/J/0xI0iHX5K6fknTFWYen19zhmnvTa875zvL0mlUYWLd2y/cl9gEAAAAAANBaDGEAAAAAAAASMIQBAAAAAABIwBAGAAAAAAAgAUMYAAAAAACABAxhAAAAAAAAEjCEAQAAAAAASMAQBgAAAAAAIAFDGAAAAAAAgAQMYQAAAAAAABIwhAEAAAAAAEjAEAYAAAAAACBB3w5hbC+2fbPtc23fYvsLto+zfaXtX9g+rOoeATQLuQMgE5kDIBOZA/SHvh3CFPaR9AFJBxRfL5d0tKQ3SXr75g+2fZrtZbaXbdT61EYBNMaUc2di5mwgcwB0h20dAJm6zpwNo2tTGwWaqt+HMLdFxPKIGJP0M0mXRURIWi5p8eYPjohzImJpRCwd1qzkVgE0xJRzZ2LmjJA5ALrDtg6ATF1nzsjgnORWgWbq9yHMxP/iGZtwe0zSUH47AFqA3AGQicwBkInMASrW70MYAAAAAACARmAIAwAAAAAAkKBvdzmLiF9JOnjC7ZO3dB8A9AK5AyATmQMgE5kD9Af2hAEAAAAAAEjAEAYAAAAAACABQxgAAAAAAIAEDGEAAAAAAAASMIQBAAAAAABIwBAGAAAAAAAgAUMYAAAAAACABAxhAAAAAAAAEjCEAQAAAAAASDBUdQNl8cCABubNS605dttvUuuN84F7pdc85n8dml7z919Ym15z79Or+ZlqwKnl/OBgar0mCkkxOlZ1GylO/80fpdfceOBj02tu96jV6TU9MpJeU5IUUU1dzFwk505F68qv/2JTes19z8j/PF79qf3Ta+7ia9JrSpIi/2eKmYkNG7Xp9oq2jZP91/8cnF5zp3X5+Xrby3ZJr/nYd9+WXrMKsZV/L9kTBgAAAAAAIAFDGAAAAAAAgAQMYQAAAAAAABIwhAEAAAAAAEjAEAYAAAAAACABQxgAAAAAAIAEDGEAAAAAAAASMIQBAAAAAABIwBAGAAAAAAAgAUMYAAAAAACABNMewthe1U0h28fa/toky0+w/dZuXhNA85E5ADKROQCykTtAuwxV3UBEXCTpoqr7ANAOZA6ATGQOgGzkDtDfuj4cyR1n2f6p7eW2T9ra8s2e+xTb19ne2/bJtj9cLD/X9tm2f2D7VtsvKZYP2P6I7ZttX2r7G+P3AWgHMgdAJjIHQDZyB2iHmewJ8yJJh0h6oqSdJV1t+wpJR25huSTJ9pGSPiTpBRHxa9vHbPa6u0k6WtIB6kxwzytqLZZ0kKRdJN0k6VMz6B1A/ZA5ADKROQCykTtAC8zkxLxHS/piRIxGxN2SvivpKVtZLkkHSjpH0vMj4tdbeN0LImIsIm6UtOuEWl8ult8l6TuTPdH2abaX2V62IdbN4K0B6EN9nTkbyRygafouc6TNckfrZ/wmAfSVvssdMgfoveyrI90paZ2kQ7fymImfbk/nxSPinIhYGhFLRzy7m/4ANEta5gyTOQBKzhxps9zRrOk+HUDz5G3rkDlAT8xkCPM9SSfZHrS9SNLTJP14K8sl6UFJz5X0XtvHTqPWlZJeXBy7uKuk6TwXQDOQOQAykTkAspE7QAvM5JwwX5V0hKTrJYWkt0TEXba3tPwASYqIu20/T9I3bZ8yxVpfkfTHkm6U9BtJ10paMYPeAdQPmQMgE5kDIBu5A7SAI6LqHqbE9ryIWGV7J3Umv0cVxy9OauHgznH4vBPyGpQU66o5TtIH7pVec9XeC9Nr/v5la9Nr7n36b9JrSpIGpr2H+oxc9eD5WrHxntyifW66mbNgYKc4fPhP8hqUFBs3pNYbt8cP56XX/PWb902veecZ+X+/j3nlbek1q3LJ6s9eExFLq+6jX0w3cyRpgXeMpw4cl9PguIq2G2/7jyXpNfc94/fpNe98wePSa+7yyWvSa0r5/4b9KC7TQ3E/2zoTTHtbxzvGU/3HeQ1W6DfnHZxec6d/3y695r1PGEyv+dh3/yi9ZhV+NHrJFjNnJnvCZPua7e0ljUh697Y2TABghsgcAJnIHADZyB2gArUZwkTEsVX3AKA9yBwAmcgcANnIHaAa2VdHAgAAAAAAaCWGMAAAAAAAAAkYwgAAAAAAACRgCAMAAAAAAJCAIQwAAAAAAEAChjAAAAAAAAAJGMIAAAAAAAAkGKq6gbLE2JjGVq7MLTowmFuvEMt/nl5zpw/tkl7z/ov3Tq85+sAD6TWrEDFadQv1F6HYuCG1pIdHUuuN2232ivSav1w4nF5z9Yr8/6cYW7s2vaYkDcyaVUldzJAtj+TmgAer2dbZ9T/npNfctGf+ts6K/SK95m677JxeU5I23Xl3bkE2dTANL9z7hvSal+x0VHrNR393XXrNyoz1TwiwJwwAAAAAAEAChjAAAAAAAAAJGMIAAAAAAAAkYAgDAAAAAACQgCEMAAAAAABAAoYwAAAAAAAACRjCAAAAAAAAJGAIAwAAAAAAkIAhDAAAAAAAQAKGMAAAAAAAAAn6Zghj+w22b7L9hap7AdAO5A6ATGQOgExkDtCfhqpuYILXSTouIn67rQfaHoqITQk9AWg2cgdAJjIHQCYyB+hDfTGEsf1vkvaS9E3b50o6pri9RtJpEXGD7TMl7V0s/7WkP6umWwBNQO4AyETmAMhE5gD9qy8OR4qI0yXdIenpkhZLui4ilkh6u6TPTnjoQepMcwkIADNC7gDIROYAyETmAP2rL/aE2czRkl4sSRHxbds72V5Q3HdRRKzd0hNtnybpNEmarbmlNwqgMbrKHTIHQJfY1gGQicwB+khf7AkzDau3dmdEnBMRSyNi6bBmZfUEoNm2mDtkDoASTH1bx7OzegLQXPx+BSTrxyHM9yS9QpJsHyvp3oh4qMqGADQeuQMgE5kDIBOZA/SRfjwc6UxJn7J9gzonjvrzatsB0AJnitwBkOdMkTkA8pwpMgfoG30zhImIxRNuvnCS+8/M6gVAO5A7ADKROQAykTlAf+rHw5EAAAAAAAAahyEMAAAAAABAAoYwAAAAAAAACRjCAAAAAAAAJGAIAwAAAAAAkIAhDAAAAAAAQAKGMAAAAAAAAAkYwgAAAAAAACRgCAMAAAAAAJBgqOoGSmPLs2alloyNm1LrjRva8zHpNTe+On9+9/ff/GJ6zc+e86T0mpI0ev+DyQVzyzWRBwY0MGdubs3H7J5ab9wVd+evMPN/8rv0msOv3Cm95tCjdk2vKUkxNpZfdG1+yaaxJNupNcfWrEmtN+63z8lfRw9862/Ta867/YD0mmMPrkivKUkeyF13VUHMNY4lD+X++hij1Wykfv2zR6fX3OMbv0qv+fxLr0+vecGS3dJrSlJEcubElu9iTxgAAAAAAIAEDGEAAAAAAAASMIQBAAAAAABIwBAGAAAAAAAgAUMYAAAAAACABAxhAAAAAAAAEjCEAQAAAAAASMAQBgAAAAAAIAFDGAAAAAAAgAQMYQAAAAAAABIwhAEAAAAAAEjAEAYAAAAAACBBbYYwti+wfY3tn9k+rep+ADQfuQMgE5kDIBOZA1RjqOoGpuGUiLjf9hxJV9v+SkTcV3VTABqN3AGQicwBkInMASpQpyHMG2yfWHz/GEn7SnpESBQT3NMkabbm5nYHoIm2mjuPyBxvl98dgKaZ3rYOuQNgZvj9CqhALYYwto+VdJykIyJije3LJc3e/HERcY6kcyRpwcBOkdgigIaZSu5MzJyFgzuTOQC61s22zkK2dQB0qbvfr3Ykc4AeqMs5YRZKeqAIiAMkHV51QwAaj9wBkInMAZCJzAEqUpchzLckDdm+SdL7JP2w4n4ANB+5AyATmQMgE5kDVKQWhyNFxHpJz666DwDtQe4AyETmAMhE5gDVqcueMAAAAAAAALXGEAYAAAAAACABQxgAAAAAAIAEDGEAAAAAAAASMIQBAAAAAABIwBAGAAAAAAAgAUMYAAAAAACABAxhAAAAAAAAEjCEAQAAAAAASMAQBgAAAAAAIMFQ1Q00ythoNXU3bkovuel3d6TXvGb14vSaK5+2T3pNSZp7wbJK6mIGZo1I+y1OLTl20/+k1hvnDz0xvWbMfyC95sYVs9Jr3n/s4vSakjQ25Pyin80v2TgDlkdGcmuuX59brzByV/4mazzmUek15905ll5zYNdF6TUlaeye+1LreRX/9zxTHhjUwPz5qTVHVzyUWm/cHuf/Jr3m2H33p9c8ffvfpde88PFPS68pSf7F7bn11m45c0gjAAAAAACABAxhAAAAAAAAEjCEAQAAAAAASMAQBgAAAAAAIAFDGAAAAAAAgAQMYQAAAAAAABIwhAEAAAAAAEjAEAYAAAAAACABQxgAAAAAAIAEDGEAAAAAAAAS9NUQxvaqbdy/2PbLs/oB0GxkDoBs5A6ATGQO0H/6aggzBYslERIAsiwWmQMg12KROwDyLBaZA6TqyyGMO86y/VPby22fVNz1PknH2P6J7TOq7BFAc5A5ALKROwAykTlA/xiquoEteJGkQyQ9UdLOkq62fYWkt0p6U0Q8b7In2T5N0mmSNFtzczoF0AQzz5zhhTmdAmiKmeeOt8vpFEATzDxzBubldAo0XF/uCSPpaElfjIjRiLhb0nclPWVbT4qIcyJiaUQsHfbs0psE0BgzzpyRIQa/AKZl5rkzwLYOgCmbeebw+xXQE/06hAEAAAAAAGiUfh3CfE/SSbYHbS+S9DRJP5a0UtL8SjsD0ERkDoBs5A6ATGQO0Cf6dQjzVUk3SLpe0rclvSUi7iqWjdq+nhNHAeghMgdANnIHQCYyB+gTfXVi3oiYV/wZkt5cfE28f6OkZ1TQGoAGInMAZCN3AGQic4D+0697wgAAAAAAADQKQxgAAAAAAIAEDGEAAAAAAAASMIQBAAAAAABIwBAGAAAAAAAgAUMYAAAAAACABAxhAAAAAAAAEjCEAQAAAAAASMAQBgAAAAAAIIEjouoeSmH7Hkm3d/n0nSXd28N2qNm+mlXV7bbmnhGxqNfNtAmZ07c1q6pLzW0jd2ZoBrlTt3WlbnWp2Z81yZwZqlnmVFWXms2r2/Pfrxo7hJkJ28siYik1qVm3ulW9V8xMW9YVPovURH9o07rSlvdKTfQzPv/UrHPdMmpyOBIAAAAAAEAChjAAAAAAAAAJGMJM7hxq/t9sr9rs9sm2P9yLmrYvt/1/7eZl+1zbt9n+SfF1yDTrbbFmgr7/maJvtGVdmXbNinLHtv/J9i22b7L9hmnW22LNkrWlJmauTetKHbZ1vjdhO+cO2xdMs94Wa5asLTUxc7X4/FdRs6LM+WPb1xaZ833b+0yz3hZrJuj7n+lUcE4YTJntVRExb8LtkyUtjYjX9+C1L5f0pohYttnycyV9LSLOm2kNAPVTUe68WtLTJZ0cEWO2d4mI38+0HoD+V0XmbPaYr0i6MCI+O9N6APpfRds5t0h6QUTcZPt1kg6LiJNnWg9Tx54w6Anbi2x/xfbVxddRxfLDbF9l+zrbP7C9f7F8ju3/KP6X+auS5lT6BgDUTom585eS3hURY5LEAAaAVP62ju0Fkp4h6YKy3wuA/ldi5oSkBcX3CyXdUfqbwSMMVd0AamWO7Z9MuL2jpIuK7z8o6V8j4vu2HyvpYkkHSrpZ0jERscn2cZLeI+nF6vySsyYiDrS9RNK1W6n7T7bfIekySW+NiPU9fVcA+lkVubO3pJNsnyjpHklviIhf9PqNAehLVW3rSNILJV0WEQ/17N0A6HdVZM5rJH3D9lpJD0k6vNdvClvHEAbTsTYiDhm/Mb67XHHzOEkH2R6/e4HteepMVz9je191pq7Dxf1Pk3S2JEXEDbZv2ELNt0m6S9KIOsfj/b+S3tWj9wOg/1WRO7MkrYuIpbZfJOlTko7p2TsC0M+qyJxxfybpEz14DwDqo4rMOUPScyLiR7bfLOn/qDOYQRKGMOiVAUmHR8S6iQuLE0t9JyJOtL1Y0uXTedGIuLP4dr3tT0t6Uw96BdAMpeSOpN9KOr/4/quSPj3DPgE0Q1mZI9s7SzpM0ok96BNAM/Q8c2wvkvTEiPhRseg/JX2rN+1iqjgnDHrlEkl/NX7DD1/FaKGk3xXfnzzh8VdIennx2IMlLZnsRW3vVvxpdXbT/WnvWgZQc6XkjjrnY3h68f0fSbqlF80CqL2yMkeSXqLOhQjWbeUxANqljMx5QNJC2/sVt58p6aaedYwpYQiDXnmDpKW2b7B9o6TTi+Xvl/Re29fpkXtefVTSPNs3qXN40TVbeN0v2F4uabmknSX9YyndA6ijsnLnfZJeXGTPe8UuugA6ysocSXqZpC+W0DOA+up55kTEJkmvlfQV29dLepWkN5f4HjAJLlENAAAAAACQgD1hAAAAAAAAEjCEAQAAAAAASMAQBgAAAAAAIAFDGAAAAAAAgAQMYQAAAAAAABIwhAEAAAAAAEjAEAYAAAAAACABQxgAAAAAAIAE/z/yiXIJefEqDQAAAABJRU5ErkJggg==\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "translate(sentence_pairs[2], plot='decoder_layer4_block2')\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 19.总结\n",
    "\n",
    "代码还有以下待完善的地方：\n",
    "- 没有实现标签平滑\n",
    "- 在训练过程中使用了teacher-forcing，即总是会将target传递到下一时间步长。更好的做法是设置一个teacher_forcing_ration\n",
    "- 在evaluate阶段的解码使用的是greedy search decode，即对于每一步，我们只需从具有最高 softmax 值的 decoder_output 中选择单词。可以尝试使用更好的beam search"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}